Evolution diagrams for debugging distributed embedded software applications

ABSTRACT

Software development methods and tools are described to generate visual representations of a system&#39;s behavior over time, called “evolution diagrams,” to aid in debugging concurrent software systems. The diagrams take advantage of the exposure provided by coordination interfaces to present more complete views of system executions, explicitly showing events, message traffic between components, etc. The display is presented at a user-selectable hierarchical level of the system design, thus enabling a programmer to work at a design layer where the problem or its effect is easily recognized.

RELATED APPLICATIONS

[0001] This application is a continuation of U.S. ProvisionalApplication No. 60/213,496 filed Jun. 23, 2000, incorporated herein inits entirety.

TECHNICAL FIELD

[0002] The present invention relates to software development tools and,more specifically, relates to creation and use of graphical displays inconnection with debugging concurrent software systems.

BACKGROUND OF THE INVENTION

[0003] A system design and programming methodology is most effectivewhen it is closely integrated and coheres tightly with its correspondingdebugging techniques. In distributed and embedded system methodologies,the relationship between debugging approaches and design methodologieshas traditionally been one-sided in favor of the design and programmingmethodologies. Design and programming methodologies are typicallydeveloped without any consideration for the debugging techniques thatwill later be applied to software systems designed using that design andprogramming methodology. While these typical debugging approachesattempt to exploit features provided by the design and programmingmethodologies, the debugging techniques will normally have little or noimpact on what the design and programming features are in the firstplace. This lack of input from debugging approaches to design andprogramming methodologies serves to maintain the role of debugging as anafterthought, even though in a typical system design, debugging consumesa majority of the design time.

[0004] Concurrence, distributed hardware architectures, real-timeconstraints, and physical disparity between development and executionenvironments complicate the debugging of complex embedded systems.Researchers have noted that designers are most productive in debuggingsystems when they have a robust mental model of the systems' correctbehavior. The preceding factors make it both more important fordesigners to use robust mental models and more difficult to form andmaintain them. Distributed systems have no system-wide schedule ofevents, and the interleaving of coherent event sequences from individualcomponents can be counter-intuitive and often surprising. Intercomponentrelationships are complex, and the bookkeeping required to trackdebugging information can be phenomenal. Debugging tools that complementhuman ability, assuming tasks that require bookkeeping rather thancreativity or intuition, are an absolute necessity. These tools mustalso present tracked information in a form that designers can quicklyunderstand.

SUMMARY OF THE INVENTION

[0005] Evolution diagrams can be used to graphically display operationof a distributed or concurrent software system to a programmer to aid indebugging. A preferred diagram explicitly shows selected events, messagetraffic between software components, changes in component behavior, andcorrelations between local behavior changes. In one embodiment of such adisplay, each selected component is represented by a correspondingtrace, interface states are displayed by corresponding traces (forexample, extending generally parallel to the component traces), andevents are made explicit by graphical symbols spanning the spaceaffected by the event. Messages, represented by arrows, present explicitdependencies, while indirect or implicit dependencies such as thoseimplemented by coordinator constraints and actions, are indicated bydiagonal lines or the like between the relevant events. See page 78.

[0006] “Selective focus,” similar to scrolling the display over time(not real time but “consistent cut” time) enables examination of aparticular event or sequence of interest to the programmer. Importantly,the user can select a hierarchical level or layer at which to examine anevolution diagram. This powerful concept allows studying the executionat the design layer where the problem or its effect is recognized;thereby identifying the specific component that requires modification.Thus debugging of complex concurrent systems is made relatively fast andeasy, and initial identification of the offending component or interfacecan be done without delving “into the code”.

[0007] The present invention leverages a coordination-centric designmethodology to improve complex embedded system debugging. We draw upon:

[0008] The behavioral exposure provided by coordination interfaces togenerate rich temporal visualizations of system behavior;

[0009] The hierarchical system models to maximize the utility ofcomputer/designer Bandwidth; and

[0010] The formal relationships between design elements to examine andabstract behavior.

[0011] Additional aspects and advantages of this invention will beapparent from the following detailed description of preferredembodiments thereof, which proceeds with reference to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 is a component of a distributed software system.

[0013]FIG. 2 is the component of FIG. 1 further having a set ofcoordination interfaces.

[0014]FIG. 3A is a prior art round-robin resource allocation protocolwith a centralized controller.

[0015]FIG. 3B is a prior art round-robin resource allocation protocolimplementing a token passing scheme.

[0016]FIG. 4A is a detailed view of a component and a coordinationinterface connected to the component for use in round-robin resourceallocation in accordance with the present invention.

[0017]FIG. 4B depicts a round-robin coordinator in accordance with thepresent invention.

[0018]FIG. 5 shows several typical ports for use in a coordinationinterface in accordance with the present invention.

[0019]FIG. 6A is a unidirectional data transfer coordinator inaccordance with the present invention.

[0020]FIG. 6B is a bidirectional data transfer coordinator in accordancewith the present invention.

[0021]FIG. 6C is a state unification coordinator in accordance with thepresent invention.

[0022]FIG. 6D is a control state mutex coordinator in accordance withthe present invention.

[0023]FIG. 7 is a system for implementing subsumption resourceallocation having components, a shared resource, and a subsumptioncoordinator.

[0024]FIG. 8 is a barrier synchronization coordinator in accordance withthe present invention.

[0025]FIG. 9 is a rendezvous coordinator in accordance with the presentinvention.

[0026]FIG. 10 depicts a dedicated RPC system having a client, a server,and a dedicated RPC coordinator coordinating the activities of theclient and the server.

[0027]FIG. 11 is a compound coordinator with both preemption andround-robin coordination for controlling the access of a set ofcomponents to a shared resource.

[0028]FIG. 12A is software system with two data transfer coordinators,each having constant message consumption and generation rules and eachconnected to a separate data-generating component and connected to thesame data-receiving component.

[0029]FIG. 12B is the software system of FIG. 12A in which the two datatransfer coordinators have been replaced with a merged data transfercoordinator.

[0030]FIG. 13 is a system implementing a first come, first servedresource allocation protocol in accordance with the present invention.

[0031]FIG. 14 is a system implementing a multiclient RPC coordinationprotocol formed by combining the first come, first served protocol ofFIG. 13 with the dedicated RPC coordinator of FIG. 10.

[0032]FIG. 15 depicts a large system in which the coordination-centricdesign methodology can be employed having a wireless device interactingwith a cellular network.

[0033]FIG. 16 shows a top-level view of the behavior and components fora system for a cell phone.

[0034]FIG. 17A is a detailed view of a GUI component of the cell phoneof FIG. 16.

[0035]FIG. 17B is a detailed view of a call log component of the cellphone of FIG. 16.

[0036]FIG. 18A is a detailed view of a voice subsystem component of thecell phone of FIG. 16.

[0037]FIG. 18B is a detailed view of a connection component of the cellphone of FIG. 16.

[0038]FIG. 19 depicts the coordination layers between a wireless deviceand a base station, and between the base station and a switching center,of FIG. 15.

[0039]FIG. 20 depicts a cell phone call management component, a masterswitching center call management component, and a call managementcoordinator connecting the respective call management components.

[0040]FIG. 21A is a detailed view of a transport component of theconnection component of FIG. 18B.

[0041]FIG. 21B is a CDMA data modulator of the transport component ofFIG. 18B.

[0042]FIG. 22 is a detailed view of a typical TDMA and a typical CDMAsignal for the cell phone of FIG. 16.

[0043]FIG. 23A is a LCD touch screen component for a Web browser GUI fora wireless device.

[0044]FIG. 23B is a Web page formatter component for the Web browser GUIfor the wireless device.

[0045]FIG. 24A is a completed GUI system for a handheld Web browser.

[0046]FIG. 24B shows the GUI system for the handheld Web browsercombined with the connection subsystem of FIG. 18B in order to accessthe cellular network of FIG. 15.

[0047]FIG. 25 is a typical space/time diagram with space represented ona vertical axis and time represented on a horizontal axis.

[0048]FIG. 26 is a space/time diagram depicting a set of system eventsand two different observations of those system events.

[0049]FIG. 27 is a space/time diagram depicting a set of system eventsand an ideal observation of the events taken by a real-time observer.

[0050]FIG. 28 is a space/time diagram depicting two different yet validobservations of a system execution.

[0051]FIG. 29 is a space/time diagram depicting a system execution andan observation of that execution take by a discrete lamport observer.

[0052]FIG. 30 is a space/time diagram depicting a set of events thateach include a lamport time stamp.

[0053]FIG. 31 is a space/time diagram illustrating the insufficiency ofscalar timestamps to characterize causality between events.

[0054]FIG. 32 is a space/time diagram depicting a set of system eventsthat each a vector time stamp.

[0055]FIG. 33 depicts a display from a partial order event tracer(POET).

[0056]FIG. 34 is a space/time diagram depicting two compound events thatare neither causal nor concurrent.

[0057]FIG. 35 is a POET display of two convex event clusters.

[0058]FIG. 36 is a basis for distributed event environments (BEE)abstraction facility for a single client.

[0059]FIG. 37 is a hierarchical tree construction of process clusters.

[0060]FIG. 38A depicts a qualitative measure of cohesion and couplingbetween a set of process clusters that have heavy communication or areinstantiated from the same source code.

[0061]FIG. 38B depicts a qualitative measure of cohesion and couplingbetween a set of process clusters that do not have heavy communicationor are not instances of the same source code.

[0062]FIG. 38C depicts a qualitative measure of cohesion and couplingbetween an alternative set of process clusters that have heavycommunication or are instantiated from the same source code.

[0063]FIG. 39 depicts a consistent and an inconsistent cut of a systemexecution on a space/time diagram.

[0064]FIG. 40A is a space/time diagram depicting a system execution.

[0065]FIG. 40B is a lattice representing all possible consistent cuts ofthe space/time diagram of FIG. 40A.

[0066]FIG. 40C is a graphical representation of the possible consistentcuts of FIG. 40B.

[0067]FIG. 41A is a space/time diagram depicting a system execution.

[0068]FIG. 41B is the space/time diagram of FIG. 41A after performing aglobal-step.

[0069]FIG. 41C is the space/time diagram of FIG. 41A after performing astep-over.

[0070]FIG. 41D is the space/time diagram of FIG. 41A after performing astep-in.

[0071]FIG. 42 is a space/time diagram depicting a system that is subjectto a domino effect whenever the system is rolled back in time to acheckpoint.

[0072]FIG. 43 is a coordinator evolution diagram for an RPC interaction.

[0073]FIG. 44 illustrates top-level event types useful for evolutiondiagrams.

[0074]FIG. 45 illustrates primitive state types useful for evolutiondiagrams.

[0075]FIG. 46 is an evolution diagram of mobile unit call initiation.

[0076] FIGS. 47A-47B are evolution diagrams illustrating local controlfailure and distributed control failure, respectively, of the callinitiation transaction.

[0077]FIG. 48 is an expanded view of the voice subsystem aspect of anexecution diagram illustrating selective focus in debugging.

[0078]FIG. 49 illustrates source code for three components highlightingthe sections of code implicated in suspect behavior.

[0079]FIG. 50 shows an evolution diagram illustrating a bug thatmanifests itself in interactions among the several components of FIG.49.

[0080] FIGS. 51A-51B illustrate a token-preemptor coordinator mapped toa distributed architecture that causes unanticipated delays.

[0081] FIGS. 52A-52B are evolution diagrams illustrating modelsimulation and actual execution, respectively, of the combinedtoken-ring/preempt coordinator of FIG. 51.

[0082] FIGS. 53A-53B are conceptual diagrams illustrating a designhierarchy and a designer's desired view, respectively, of the GUI model.

[0083]FIG. 54 illustrates component, event and state clusters inevolution diagrams to reduce clutter.

[0084]FIG. 55A illustrates a portion of an evolution diagram beforefiltering.

[0085]FIG. 55B illustrate the diagram of FIG. 55A after filtering tohide behavioral clutter.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0086] Coordination-Centric Software Design

[0087]FIG. 1 is an example of a component 100, which is the basicsoftware element within the coordination-centric design framework, inaccordance with the present invention. With reference to FIG. 1,component 100 contains a set of modes 102. Each mode 102 corresponds toa specific behavior associated with component 100. Each mode 102 caneither be active or inactive, respectively enabling or disabling thebehavior corresponding to that mode 102. Modes 102 can make theconditional aspects of the behavior of component 100 explicit. Thebehavior of component 100 is encapsulated in a set of actions 104, whichare discrete, event-triggered behavioral elements within thecoordination-centric design methodology. Component 100 can be copied andthe copies of component 100 can be modified, providing the code-sharingbenefits of inheritance.

[0088] Actions 104 are enabled and disabled by modes 102, and hence canbe thought of as effectively being properties of modes 102. An event(not shown) is an instantaneous condition, such as a timer tick, a datadeparture or arrival, or a mode change. Actions 104 can activate anddeactivate modes 102, thereby selecting the future behavior of component100. This is similar to actor languages, in which methods are allowed toreplace an object's behavior.

[0089] In coordination-centric design, however, all possible behaviorsmust be identified and encapsulated before runtime. For example, adesigner building a user interface component for a cell phone mightdefine one mode for looking up numbers in an address book (in which theuser interface behavior is to display complete address book entries informatted text) and another mode for displaying the status of the phone(in which the user interface behavior is to graphically display thesignal power and the battery levels of the phone). The designer mustdefine both the modes and the actions for the given behaviors wellbefore the component can be executed.

[0090]FIG. 2 is component 100 further including a first coordinationinterface 200, a second coordination interface 202, and a thirdcoordination interface 204. Coordination-centric design's components 100provide the code-sharing capability of object-oriented inheritancethrough copying. Another aspect of object-oriented inheritance ispolymorphism through shared interfaces. In object-oriented languages, anobject's interface is defined by its methods. Althoughcoordination-centric design's actions 104 are similar to methods inobject-oriented languages, they do not define the interface forcomponent 100. Components interact through explicit and separatecoordination interfaces, in this figure coordination interfaces 200,202, and 204. The shape of coordination interfaces 200, 202, and 204determines the ways in which component 100 may be connected within asoftware system. The way coordination interfaces 200, 202, and 204 areconnected to modes 102 and actions 104 within component 100 determineshow the behavior of component 100 can be managed within a system.Systemwide behavior is managed through coordinators (see FIG. 4B andsubsequent).

[0091] For our approach to be effective, several factors in the designof software elements must coincide: packaging, internal organization,and how elements coordinate their behavior. Although these are oftentreated as independent issues, conflicts among them can exacerbatedebugging. We handle them in a unified framework that separates theinternal activity from the external relationship of component 100. Thislets designers build more modular components and encourages them tospecify distributable versions of coordination protocols. Components canbe reused in a variety of contexts, both distributed, and singleprocessor 1.

[0092] 1. Introduction to Coordination

[0093] Within this application, coordination refers to the predeterminedways by which components interact. Consider a common coordinationactivity: resource allocation. One simple protocol for this isround-robin: participants are lined up, and the resource is given toeach participant in turn. After the last participant is served, theresource is given back to the first. There is a resource-schedulingperiod during which each participant gets the resource exactly once,whether or not it is needed.

[0094]FIG. 3A is prior art round-robin resource allocation protocol witha centralized controller 300, which keeps track of and distributes theshared resource (not shown) to each of software elements 302, 304, 306,308, and 310 in turn. With reference to FIG. 3A, controller 300 alonedetermines which software element 302, 304, 306, 308, or 310 iscurrently allowed to use the resource and which has it next. Thisimplementation of a round-robin protocol permits software elements 302,304, 306, 308, and 310 to be modular, because only controller 300 keepstrack of the software elements. Unfortunately, when this implementationis implemented on a distributed architecture (not shown), controller 300must typically be placed on a single processing element (not shown). Asa result, all coordination requests must go through that processingelement, which can cause a communication performance bottleneck. Forexample, consider the situation in which software elements 304 and 306are implemented on a first processing element (not shown) and controller300 is implemented on a second processing element. Software element 304releases the shared resource and must send a message indicating this tocontroller 300. Controller 300 must then send a message to softwareelement 306 to inform software element 306 that it now has the right tothe shared resource. If the communication channel between the firstprocessing element and the second processing element is in use or thesecond processing element is busy, then the shared resource must remainidle, even though both the current resource holder and the next resourceholder (software elements 304 and 306 respectively) are implemented onthe first processing element (not shown). The shared resource musttypically remain idle until communication can take place and controller300 can respond. This is an inefficient way to control access to ashared resource.

[0095]FIG. 3B is a prior art round-robin resource allocation protocolimplementing a token passing scheme. With reference to FIG. 3B, thissystem consists of a shared resource 311 and a set of software elements312, 314, 316, 318, 320, and 322. In this system a logical token 324symbolizes the right to access resource 311, i.e., when a softwareelement holds token 324, it has the right to access resource 311. Whenone of software elements 312, 314, 316, 318, 320, or 322 finishes withresource 311, it passes token 324, and with token 324 the access right,to a successor. This implementation can be distributed without acentralized controller, but as shown in FIG. 3B, this is less modular,because it requires each software element in the set to keep track of asuccessor.

[0096] Not only must software elements 312, 314, 316, 318, 320, and 322keep track of successors, but each must implement a potentiallycomplicated and error-prone protocol for transferring token 324 to itssuccessor. Bugs can cause token 324 to be lost or introduce multipletokens 324. Since there is no formal connection between the physicalsystem and complete topology maps (diagrams that show how each softwareelement is connected to others within the system), some softwareelements might erroneously be serviced more than once per cycle, whileothers are completely neglected. However, these bugs can be extremelydifficult to track after the system is completed. The protocol isentangled with the functionality of each software element, and it isdifficult to separate the two for debugging purposes. Furthermore, if afew of the software elements are located on the same machine,performance of the implementation can be poor. The entangling ofcomputation and coordination requires intrusive modification to optimizethe system.

[0097] 2. Coordination-Centric Design's Approach to Coordination

[0098] The coordination-centric design methodology provides anencapsulating formalism for coordination. Components such as component100 interact using coordination interfaces, such as first, second, andthird coordination interfaces 200, 202, and 204, respectively.Coordination interfaces preserve component modularity while exposing anyparts of a component that participate in coordination. This technique ofconnecting components provides polymorphism in a similar fashion tosubtyping in object-oriented languages.

[0099]FIG. 4A is a detailed view of a component 400 and a resourceaccess coordination interface 402 connected to component 400 for use ina round-robin coordination protocol in accordance with the presentinvention. With reference to FIG. 4A, resource access coordinationinterface 402 facilitates implementation of a round-robin protocol thatis similar to the token-passing round-robin protocol described above.Resource access coordination interface 402 has a single bit of controlstate, called access, which is shown as an arbitrated control port 404that indicates whether or not component 400 is holding a virtual token(not shown). Component 400 can only use a send message port 406 onaccess coordination interface 402 when arbitrated control port 404 istrue. Access coordination interface 402 further has a receive messageport 408.

[0100]FIG. 4B show a round-robin coordinator 410 in accordance with thepresent invention. With reference to FIG. 4B, round-robin coordinator410 has a set of coordinator coordination interfaces 412 for connectingto a set of components 400. Each component 400 includes a resourceaccess coordination interface 402. Each coordinator coordinationinterface 412 has a coordinator arbitrated control port 414, an incomingsend message port 416 and an outgoing receive message port 418.Coordinator coordination interface 412 in complimentary to resourceaccess coordination interface 402, and vice versa, because the ports onthe two interfaces are compatible and can function to transferinformation between the two interfaces.

[0101] The round-robin protocol requires round-robin coordinator 410 tomanage the coordination topology. Round-robin coordinator 410 is aninstance of more general abstractions called coordination classes, inwhich coordination classes define specific coordination protocols and acoordinator is a specific implementation of the coordination class.Round-robin coordinator 410 contains all information about howcomponents 400 are supposed to coordinate. Although round-robincoordinator 410 can have a distributed implementation, no component 400is required to keep references to any other component 400 (unlike thedistributed round-robin implementation shown in FIG. 3B). All requiredreferences are maintained by round-robin coordinator 410 itself, andcomponents 400 do not even need to know that they are coordinatingthrough round-robin. Resource access coordination interface 402 can beused with any coordinator that provides the appropriate complementaryinterface. A coordinator's design is independent of whether it isimplemented on a distributed platform or on a monolithic singleprocessor platform.

[0102] 3. Coordination Interfaces

[0103] Coordination interfaces are used to connect components tocoordinators. They are also the principle key to a variety of usefulruntime debugging techniques. Coordination interfaces support componentmodularity by exposing all parts of the component that participate inthe coordination protocol. Ports are elements of coordinationinterfaces, as are guarantees and requirements, each of which will bedescribed in turn.

[0104] A. Ports

[0105] A port is a primitive connection point for interconnectingcomponents. Each port is a five-tuple (T; A; Q; D; R) in which:

[0106] T represents the data type of the port. T can be one of int,boolean, char, byte, float, double, or cluster, in which clusterrepresents a cluster of data types (e.g., an int followed by a floatfollowed by two bytes).

[0107] A is a boolean value that is true if the port is arbitrated andfalse otherwise.

[0108] Q is an integer greater than zero that represents logical queuedepth for a port.

[0109] D is one of in, out, inout, or custom and represents thedirection data flows with respect to the port.

[0110] R is one of discard-on-read, discard-on-transfer, or hold andrepresents the policy for data removal on the port. Discard-on-readindicates that data is removed immediately after it is read (and anydata in the logical queue are shifted), discard-on-transfer indicatesthat data is removed from a port immediately after being transferred toanother port, and hold indicates that data should be held until it isoverwritten by another value. Hold is subject to arbitration.

[0111] Custom directionality allows designers to specify ports thataccept or generate only certain specific values. For example, a designermay want a port that allows other components to activate, but notdeactivate, a mode. While many combinations of port attributes arepossible, we normally encounter only a few. The three most common aremessage ports (output or input), state ports (output, input, or both;sometimes arbitrated), and control ports (a type of state port). FIG. 5illustrates the visual syntax used for several common ports throughoutthis application. With reference to FIG. 5, this figure depicts anexported state port 502, an imported state port 504, an arbitrated stateport 506, an output data port 508, and an input data port 510.

[0112] 1. Message Ports

[0113] Message ports (output and input) data ports 508 and 510respectively) are either send (T; false; 1; out; discard-on-transfer) orreceive (T; false; Q; in; discard-on-read). Their function is totransfer data between components. Data passed to a send port istransferred immediately to the corresponding receive port, thus itcannot be retrieved from the send port later. Receive data ports canhave queues of various depths. Data arrivals on these ports arefrequently used to trigger and pass data parameters into actions. Valuesremain on receive ports until they are read.

[0114] 2. State Ports

[0115] State ports take one of three forms:

[0116] 1. (T; false; 1; out; hold)

[0117] 2. (T; false; 1; in; hold)

[0118] 3. (T; true; 1; inout; hold)

[0119] State ports, such as exported state port 502, imported state port504, and arbitrated state port 506, hold persistent values, and thevalue assigned to a state port may be arbitrated. This means that,unlike message ports, values remain on the state ports until changed.When multiple software elements simultaneously attempt to alter thevalue of arbitrated state port 506, the final value is determined basedon arbitration rules provided by the designer through an arbitrationcoordinator (not shown).

[0120] State ports transfer variable values between scopes. Incoordination-centric design, all variables referenced by a component arelocal to that component, and these variables must be explicitly declaredin the component's scope. Variables can, however, be bound to stateports that are connected to other components. In this way a variablevalue can be transferred between components and the variable valueachieves the system-level effect of a multivariable.

[0121] 3. Control Ports

[0122] Control ports are similar to state ports, but a control port islimited to having the boolean data type. Control ports are typicallybound to modes. Actions interact with a control port indirectly, bysetting and responding to the values of a mode that is bound to thecontrol port.

[0123] For example, arbitrated control port 404 shown in FIG. 4A is acontrol port that can be bound to a mode (not shown) containing allactions that send data on a shared channel. When arbitrated control port404 is false, the mode is inactive, disabling all actions that send dataon the channel.

[0124] B. Guarantees

[0125] Guarantees are formal declarations of invariant properties of acoordination interface. There can be several types of guarantees, suchas timing guarantees between events, guarantees between control state(e.g., state A and state B are guaranteed to be mutually exclusive),etc. Although a coordination interface's guarantees reflect propertiesof the component to which the coordination interface is connected, theguarantees are not physically bound to any internal portions of thecomponent. Guarantees can often be certified through static analysis ofthe software system. Guarantees are meant to cache various propertiesthat are inherent in a component or a coordinator in order to simplifystatic analysis of the software system.

[0126] A guarantee is a promise provided by a coordination interface.The guarantee takes the form of a predicate promised to be invariant. Inprinciple, guarantees can include any type of predicate (e.g., x>3, inwhich x is an integer valued state port, or t_(ea)−t_(eb)<2 ms).Throughout the remainder of this application, guarantees will be onlyevent-ordering guarantees (guarantees that specify acceptable orders ofevents) or control-relationship guarantees (guarantees pertaining toacceptable relative component behaviors).

[0127] C. Requirements

[0128] A requirement is a formal declaration of the properties necessaryfor correct software system functionality. An example of a requirementis a required response time for a coordination interface—the number ofmessages that must have arrived at the coordination interface before thecoordination interface can transmit, or fire, the messages. When twocoordination interfaces are bound together, the requirements of thefirst coordination interface must be conservatively matched by theguarantees of the second coordination interface (e.g., x<7 as aguarantee conservatively matches x<8 as a requirement). As withguarantees, requirements are not physically bound to anything within thecomponent itself. Guarantees can often be verified to be sufficient forthe correct operation of the software system in which the component isused. In sum, a requirement is a predicate on a first coordinationinterface that must be conservatively matched with a guarantee on acomplementary second coordination interface.

[0129] D. Conclusion Regarding Coordination Interfaces

[0130] A coordination interface is a four-tuple (P; G; R; I) in which:

[0131] P is a set of named ports.

[0132] G is a set of named guarantees provided by the interface.

[0133] R is a set of named requirements that must be matched byguarantees of connected interfaces.

[0134] I is a set of named coordination interfaces.

[0135] As this definition shows, coordination interfaces are recursive.Coordinator coordination interface 412, shown in FIG. 4B, used forround-robin coordination is called AccessInterface and is defined inTable 1. Constituent Value Ports P = {access: StatePort, s:outMessagePort, r: inMessagePort} Guarantees G ={

access =>

s.gen } Requirement R = Ø interfaces I = Ø

[0136] Related to coordination interfaces is a recursive coordinationinterface descriptor, which is a five-tuple (P_(a); G_(a); R_(a); I_(d);N_(d)) in which:

[0137] P_(a) is a set of abstract ports, which are ports that may beincomplete in their attributes (i.e., they do not yet have a datatype).

[0138] G_(a) is a set of abstract guarantees, which are guaranteesbetween abstract ports.

[0139] R_(a) is a set of abstract requirements, which are requirementsbetween abstract ports.

[0140] I_(d) is a set of coordination interface descriptors.

[0141] N_(d) is an element of Q×Q, where Q={∞}∪Z+ and Z+ denotes the setof positive integers. N_(d) indicates the number or range of numbers ofpermissible interfaces.

[0142] Allowing coordination interfaces to contain other coordinationinterfaces is a powerful feature. It lets designers use commoncoordination interfaces as complex ports within other coordinationinterfaces. For example, the basic message ports described above arenonblocking, but we can build a blocking coordination interface (notshown) that serves as a blocking port by combining a wait state portwith a message port.

[0143] 4. Coordinators

[0144] A coordinator provides the concrete representations ofintercomponent aspects of a coordination protocol. Coordinators allow avariety of static analysis debugging methodologies for software systemscreated with the coordination-centric design methodology. A coordinatorcontains a set of coordination interfaces and defines the relationshipsthe coordination interfaces. The coordination interfaces complement thecomponent coordination interfaces provided by components operatingwithin the protocol. Through matched interface pairs, coordinatorseffectively describe connections between message ports, correlationsbetween control states, and transactions between components.

[0145] For example, round-robin coordinator 410, shown in FIG. 4B, mustensure that only one component 400 has its component control port 404'svalue, or its access bit, set to true. Round-robin coordinator 410 mustfurther ensure that the correct component 400 has its component controlport 404 set to true for the chosen sequence. This section presentsformal definitions of the parts that comprise coordinators: modes,actions, bindings, action triples, and constraints. These definitionsculminate in a formal definition of coordinators.

[0146] A. Modes

[0147] A mode is a boolean value that can be used as a guard on anaction. In a coordinator, the mode is most often bound to a control portin a coordination interface for the coordinator. For example, inround-robin coordinator 410, the modes of concern are bound to acoordinator control port 414 of each coordinator coordination interface412.

[0148] B. Actions

[0149] An action is a primitive behavioral element that can:

[0150] Respond to events.

[0151] Generate events.

[0152] Change modes.

[0153] Actions can range in complexity from simple operations up tocomplicated pieces of source code. An action in a coordinator is calleda transparent action because the effects of the action can beprecomputed and the internals of the action are completely exposed tothe coordination-centric design tools.

[0154] C. Bindings

[0155] Bindings connect input ports to output ports, control ports tomodes, state ports to variables, and message ports to events. Bindingsare transparent and passive. Bindings are simply conduits for eventnotification and data transfer. When used for event notification,bindings are called triggers.

[0156] D. Action Triples

[0157] To be executed, an action must be enabled by a mode and triggeredby an event. The combination of a mode, trigger, and action is referredto as an action triple, which is a triple (m; t; a) in which:

[0158] m is a mode.

[0159] t is a trigger.

[0160] a is an action.

[0161] The trigger is a reference to an event type, but it can be usedto pass data into the action. Action triples are written: mode : trigger: action.

[0162] A coordinator's actions are usually either pure control, in whichboth the trigger and action performed affect only control state, or puredata, in which both the trigger and action performed occur in the datadomain. In the case of round-robin coordinator 410, the following set ofactions is responsible for maintaining the appropriate state:

access_(i) : −access_(i) : +access_((i+1)mod n)

[0163] The symbol “+” signifies a mode's activation edge (i.e., theevent associated with the mode becoming true), and the symbol “−”signifies its deactivation edge. When any coordinator coordinationinterface 412 deactivates its arbitrated control port 404's, access bit,the access bit of the next coordinator coordination interface 412 isautomatically activated.

[0164] E. Constraints

[0165] In this dissertation, constraints are boolean relationshipsbetween control ports. They take the form:

Condition

Effect

[0166] This essentially means that the Condition (on the left side ofthe arrow) being true implies that Effect (on the right side of thearrow) is also true. In other words, if Condition is true, then Effectshould also be true.

[0167] A constraint differs from a guarantee in that the guarantee islimited to communicating in-variant relationships between componentswithout providing a way to enforce the in-variant relationship. Theconstraint, on the other hand, is a set of instructions to the runtimesystem dealing with how to enforce certain relationships betweencomponents. When a constraint is violated, two corrective actions areavailable to the system: (1) modify the values on the left-hand side tomake the left-hand expression evaluate as false (an effect termedbackpressure) or (2) alter the right-hand side to make it true. We referto these techniques as LHM (left-hand modify) and RHM (right-handmodify). For example, given the constraint x

y and the value x^ y, with RHM semantics the runtime system must respondby disabling y or setting y to false. Thus the value of

y is set to true.

[0168] The decision of whether to use LHM, to use RHM, or even tosuspend enforcement of a constraint in certain situations candramatically affect the efficiency and predictability of the softwaresystem. Coordination-centric design does not attempt to solvesimultaneous constraints at runtime. Rather, runtime algorithms uselocal ordered constraint solutions. This, however, can result in someconstraints being violated and is discussed further below.

[0169] Round-robin coordinator 410 has a set of safety constraints toensure that there is never more than one token in the system:

access_(i)

∀_(j≠i)

access_(j)

[0170] The above equation translates roughly as access_(i) implies notaccess_(j) for the set of all access_(j) where j is not equal to i. Eventhis simple constraint system can cause problems with local resolutionsemantics (as are LHM and RHM). If the runtime system attempted to fixall constraints simultaneously, all access modes would be shut down. Ifthey were fixed one at a time, however, any duplicate tokens would beerased on the first pass, satisfying all other constraints and leaving asingle token in the system.

[0171] Since high-level protocols can be built from combinations oflower-level protocols, coordinators can be hierarchically composed. Acoordinator is a six-tuple (I; M; B; N; A; X) in which:

[0172] I is a set of coordination interfaces.

[0173] M is a set of modes.

[0174] B is a set of bindings between interface elements (e.g., controlports and message ports) and internal elements (e.g., modes andtriggers).

[0175] N is a set of constraints between interface elements.

[0176] A is a set of action triples for the coordinator.

[0177] X is a set of subcoordinators.

[0178]FIGS. 6A, 6B, 6C, and 6D show a few simple coordinatorshighlighting the bindings and constraints of the respectivecoordinators. With reference to FIG. 6A, a unidirectional data transfercoordinator 600 transfers data in one direction between two components(not shown) by connecting incoming receive message port 408 to outgoingreceive message port 418 with a binding 602. With reference to FIG. 6B,bidirectional data transfer coordinator 604 transfers data back andforth between two components (not shown) by connecting incoming receivemessage port 408 to outgoing receive message port 418 with binding 602and connecting send message port 406 to incoming send message port 416with a second binding 602. Unidirectional data transfer coordinator 600and bidirectional data transfer coordinator 604 simply move data fromone message port to another. Thus each coordinator consists of bindingsbetween corresponding ports on separate coordination interfaces.

[0179] With reference to FIG. 6C, state unification coordinator 606ensures that a state port a 608 and a state port b 610 are always set tothe same value. State unification coordinator 606 connects state port a608 to state port b 610 with binding 602. With reference to FIG. 6D,control state mutex coordinator 612 has a first constraint 618 and asecond constraint 620 as follows:

c

d and  (1)

d

c.  (2)

[0180] Constraints 618 and 620 can be restated as follows:

[0181] (1) A state port c 614 having a true value implies that a stateport d 616 has a false value, and

[0182] (2) State port d 616 having a true value implies that state portc 614 has a false value.

[0183] A coordinator has two types of coordination interfaces: upinterfaces that connect the coordinator to a second coordinator, whichis at a higher level of design hierarchy and down interfaces thatconnect the coordinator either to a component or to a third coordinator,which is at a lower level of design hierarchy. Down interfaces havenames preceded with “˜”. Round-robin coordinator 410 has six downcoordination interfaces (previously referred to as coordinatorcoordination interface 412), with constraints that make the turning offof any coordinator control port 414 (also referred to as access controlport) turn on the coordinator control port 414 of the next coordinatorcoordination interface 412 in line. Table 2 presents all constituents ofthe round-robin coordinator. Constituent Value Coordination interfaces I= AccessInterface₁₋₆ Modes M = access₁₋₆ Bindings B =∀_(1≦i≦6)(˜AccessInterface_(i),access,access_(i)) ∪ Constraints N =∀_(1≦i≦6)(∀_((1≦j≦6)^ (i≠j))access_(i) =>

access_(j)) Actions A = ∀_(1≦i≦6)access_(i): −access_(i):+access_((i+1)) mod 6 Subcoordinators X = Ø

[0184] This tuple describes an implementation of a round-robincoordination protocol for a particular system with six components, asshown in round-robin coordinator 410. We use a coordination class todescribe a general coordination protocol that may not have a fixednumber of coordinator coordination interfaces. The coordination class isa six-tuple (Ic; Mc; Bc; Nc; Ac; Xc) in which:

[0185] Ic is a set of coordination interface descriptors in which eachdescriptor provides a type of coordination interface and specifies thenumber of such interfaces allowed within the coordination class.

[0186] Mc is a set of abstract modes that supplies appropriate modeswhen a coordination class is instantiated with a fixed number ofcoordinator coordination interfaces.

[0187] Bc is a set of abstract bindings that forms appropriate bindingsbetween elements when the coordination class is instantiated.

[0188] Nc is a set of abstract constraints that ensures appropriateconstraints between coordination interface elements are in place asspecified at instantiation.

[0189] Ac is a set of abstract action triples for the coordinator.

[0190] Xc is a set of coordination classes (hierarchy).

[0191] While a coordinator describes coordination protocol for aparticular application, it requires many aspects, such as the number ofcoordination interfaces and datatypes, to be fixed. Coordination classesdescribe protocols across many applications. The use of the coordinationinterface descriptors instead of coordination interfaces letscoordination classes keep the number of interfaces and datatypesundetermined until a particular coordinator is instantiated. Forexample, a round-robin coordinator contains a fixed number ofcoordinator coordination interfaces with specific bindings andconstraints between the message and state ports on the fixed number ofcoordinator coordination interfaces. A round-robin coordination classcontains descriptors for the coordinator coordination interface type,without stating how many coordinator coordination interfaces, andinstructions for building bindings and constraints between ports on thecoordinator coordination interfaces when a particular round-robincoordinator is created.

[0192] 5. Components

[0193] A component is a six-tuple (I; A; M; V; S; X) in which:

[0194] I is a set of coordination interfaces.

[0195] A is a set of action triples.

[0196] M is a set of modes.

[0197] V is a set of typed variables.

[0198] S is a set of subcomponents.

[0199] X is a set of coordinators used to connect the subcomponents toeach other and to the coordination interfaces.

[0200] Actions within a coordinator are fairly regular, and hence alarge number of actions can be described with a few simple expressions.However, actions within a component are frequently diverse and canrequire distinct definitions for each individual action. Typically acomponent's action triples are represented with a table that has threecolumns: one for the mode, one for the trigger, and one for the actioncode. Table 3 shows some example actions from a component that can useround-robin coordination. Mode Trigger Action Access tickAccessInterface.s.send(“Test message”); −access;

access tick waitCount ++;

[0201] A component resembles a coordinator in several ways (for example,the modes and coordination interfaces in each are virtually the same).Components can have internal coordinators, and because of the internalcoordinators, components do not always require either bindings orconstraints. In the following subsections, various aspects of componentsare described in greater detail. Theses aspects of components includevariable scope, action transparency, and execution semantics for systemsof actions.

[0202] A. Variable Scope

[0203] To enhance a component's modularity, all variables accessed by anaction within the component are either local to the action, local to theimmediate parent component of the action, or accessed by the immediateparent component of the action via state ports in one of the parentcomponent's coordination interfaces. For a component's variables to beavailable to a hierarchical child component, they must be exported bythe component and then imported by the child of the component.

[0204] B. Action Transparency

[0205] An action within a component can be either a transparent actionor an opaque action. Transparent and opaque actions each have differentinvocation semantics. The internal properties, i.e. control structures,variable, changes in state, operators, etc., of transparent actions arevisible to all coordination-centric design tools. The design tools canseparate, observe, and analyze all the internal properties of opaqueactions. Opaque actions are source code. Opaque actions must be executeddirectly, and looking at the internal properties of opaque actions canbe accomplished only through traditional, source-level debuggingtechniques. An opaque action must explicitly declare any mode changesand coordination interfaces that the opaque action may directly affect.

[0206] C. Action Execution

[0207] An action is triggered by an event, such as data arriving ordeparting a message port, or changes in value being applied to a stateport. An action can change the value of a state port, generate an event,and provide a way for the software system to interact with low-leveldevice drivers. Since actions typically produce events, a single triggercan be propagated through a sequence of actions.

[0208] 6. Protocols Implemented with Coordination Classes

[0209] In this section, we describe several coordinators thatindividually implement some common protocols: subsumption, barriersynchronization, rendezvous, and dedicated RPC.

[0210] A. Subsumption Protocol

[0211] A subsumption protocol is a priority-based, preemptive resourceallocation protocol commonly used in building small, autonomous robots,in which the shared resource is the robot itself.

[0212]FIG. 7 shows a set of coordination interfaces and a coordinatorfor implementing the subsumption protocol. With reference to FIG. 7, asubsumption coordinator 700 has a set of subsumption coordinatorcoordination interfaces 702, which have a subsume arbitrated coordinatorcontrol port 704 and an incoming subsume message port 706. Each subsumecomponent 708 has a subsume component coordination interface 710.Subsume component coordination interface 710 has a subsume arbitratedcomponent control port 712 and an outgoing subsume message port 714.Subsumption coordinator 700 and each subsume component 708 are connectedby their respective coordination interfaces, 702 and 710. Eachsubsumption coordinator coordination interface 702 in subsumptioncoordinator 700 is associated with a priority. Each subsume component708 has a behavior that can be applied to a robot (not shown). At anytime, any subsume component 708 can attempt to assert its behavior onthe robot. The asserted behavior coming from the subsume component 708connected to the subsumption coordinator coordination interface 702 withthe highest priority is the asserted behavior that will actually beperformed by the robot. Subsume components 708 need not know anythingabout other components in the system. In fact, each subsume component708 is designed to perform independently of whether their assertedbehavior is performed or ignored.

[0213] Subsumption coordinator 700 further has a slave coordinatorcoordination interface 716, which has an outgoing slave message port718. Outgoing slave message port 718 is connected to an incoming slavemessage port 720. Incoming slave message port 720 is part of a slavecoordination interface 722, which is connected to a slave 730. When asubsume component 708 asserts a behavior and that component has thehighest priority, subsumption coordinator 700 will control slave 730(which typically controls the robot) based on the asserted behavior.

[0214] The following constraint describes the basis of the subsumptioncoordinator 700's behavior:$\left. {subsume}_{p}\Rightarrow{\overset{p - 1}{\underset{i = 1}{}}{{subsume}_{i}}} \right.$

[0215] This means that if any subsume component 708 has a subsumearbitrated component control port 712 that has a value of true, then alllower-priority subsume arbitrated component control ports 712 are set tofalse. An important difference between round-robin and subsumption isthat in round-robin, the resource access right is transferred only whensurrendered. Therefore, round-robin coordination has cooperative releasesemantics. However, in subsumption coordination, a subsume component 708tries to obtain the resource whenever it needs to and succeeds only whenit has higher priority than any other subsume component 708 that needsthe resource at the same time. A lower-priority subsume component 708already using the resource must surrender the resource whenever ahigher-priority subsume component 708 tries to access the resource.Subsumption coordination uses preemptive release semantics, whereby eachsubsume component 708 must always be prepared to relinquish theresource.

[0216] Table 4 presents the complete tuple for the subsumptioncoordinator. Constituent Value Coordination interfaces I =(Subsume_(1−n)) ∪ (Output) Modes M = subsum_(1−n) Bindings B = ∀_(1≦i≦n)(Subsume_(i),subsume,subsume_(i)) ∪ Constraints N =∀_(1≦i≦n)(∀_(1≦j≦i))subsume_(i) =>

subsume_(j)) Actions A = Ø Subcoordinators X = Ø

[0217] B. Barrier Synchronization Protocol

[0218] Other simple types of coordination that components might engagein enforce synchronization of activities. An example is barriersynchronization, in which each component reaches a synchronization pointindependently and waits. FIG. 8 depicts a barrier synchronizationcoordinator 800. With reference to FIG. 8, barrier synchronizationcoordinator 800 has a set of barrier synchronization coordinationinterfaces 802, each of which has a coordinator arbitrated state port804, named wait. Coordinator arbitrated state port 804 is connected to acomponent arbitrated state port 806, which is part of a componentcoordination interface 808. Component coordination interface 808 isconnected to a component 810. When all components 810 reach theirrespective synchronization points, they are all released from waiting.The actions for a barrier synchronization coordinator with n interfacesare:$\underset{0 \leq i < n}{\Lambda}\quad {{wait}_{i}\quad:\quad {:\quad {\forall_{0 \leq j < n}{- {wait}_{j}}}}}$

[0219] In other words, when all wait modes (not shown) become active,each one is released. The blank between the two colons indicates thatthe trigger event is the guard condition becoming true.

[0220] C. Rendezvous Protocol

[0221] A resource allocation protocol similar to barrier synchronizationis called rendezvous. FIG. 9 depicts a rendezvous coordinator 900 inaccordance with the present invention. With reference to FIG. 9,rendezvous coordinator 900 has a rendezvous coordination interface 902,which has a rendezvous arbitrated state port 904. A set of rendezvouscomponents 906, each of which may perform different functions or havevastly different actions and modes, has a rendezvous componentcoordination interface 908, which includes a component arbitrated stateport 910. Rendezvous components 906 connect to rendezvous coordinator900 through their respective coordination interfaces, 908 and 902.Rendezvous coordinator 900 further has a rendezvous resourcecoordination interface 912, which has a rendezvous resource arbitratedstate port 914, also called available. A resource 916 has a resourcecoordination interface 918, which has a resource arbitrated state port920. Resource 916 is connected to rendezvous coordinator 900 by theircomplementary coordination interfaces, 918 and 912 respectively.

[0222] With rendezvous-style coordination, there are two types ofparticipants: resource 916 and several resource users, here rendezvouscomponents 916. When resource 916 is available, it activates itsresource arbitrated state port 920, also referred to as its availablecontrol port. If there are any waiting rendezvous components 916, onewill be matched with the resource; both participants are then released.This differs from subsumption and round-robin in that resource 916 playsan active role in the protocol by activating its available control port920.

[0223] The actions for rendezvous coordinator 900 are:

available_(i)^ wait_(j): :−available_(i), −wait_(j)

[0224] This could also be accompanied by other modes that indicate thestatus after the rendezvous. With rendezvous coordination, it isimportant that only one component at a time be released from wait mode.

[0225] D. Dedicated RPC Protocol

[0226] A coordination class that differs from those described above isdedicated RPC. FIG. 10 depicts a dedicated RPC system. With reference toFIG. 10, a dedicated RPC coordinator 1000 has an RPC server coordinationinterface 1002, which includes an RPC server imported state port 1004,an RPC server output message port 1006, and an RPC server input messageport 1008. Dedicated RPC coordinator 1000 is connected to a server 1010.Server 1010 has a server coordination interface 1012, which has a serverexported state port 1014, a server input data port 1016, and a serveroutput data port 1018. Dedicated RPC coordinator 1000 is connected toserver 1010 through their complementary coordination interfaces, 1002and 1012 respectively. Dedicated RPC coordinator 1000 further has an RPCclient coordination interface 1020, which includes an RPC clientimported state port 1022, an RPC client input message port 1024, and anRPC client output message port 1026. Dedicated RPC coordinator 1000 isconnected to a client 1028 by connecting RPC client coordinationinterface 1020 to a complementary client coordination interface 1030.Client coordination interface 1030 has a client exported state port1032, a client output message port 1034, and a client input message port1036.

[0227] The dedicated RPC protocol has a client/server protocol in whichserver 1010 is dedicated to a single client, in this case client 1028.Unlike the resource allocation protocol examples, the temporal behaviorof this protocol is the most important factor in defining it. Thefollowing transaction listing describes this temporal behavior:

[0228] Client 1028 enters blocked mode by changing the value stored atclient exported state port 1032 to true.

[0229] Client 1028 transmits an argument data message to server 1010 viaclient output message port 1034.

[0230] Server 1010 receives the argument (labeled “a”) data message viaserver input data port 1016 and enters serving mode by changing thevalue stored in server exported state port 1014 to true.

[0231] Server 1010 computes return value.

[0232] Server 1010 transmits a return (labeled “r”) message to client1020 via server output data port 1018 and exits serving mode by changingthe value stored in server exported state port 1014 to false.

[0233] Client 1028 receives the return data message via client inputmessage port 1036 and exits blocked mode by changing the value stored atclient exported state port 1032 to false.

[0234] This can be presented more concisely with an expressiondescribing causal relationships:

T_(RPC)=+client.blocked→client.transmits→

+server.serving→server.transmits→

(−server.serving∥client.receives)→−client.blocked

[0235] The transactions above describe what is supposed to happen. Otherproperties of this protocol must be described with temporal logicpredicates.

[0236] server.serving

client.blocked

[0237] server.serving

F(server.r.output)

[0238] server.a.input

F(server.serving)

[0239] The r in server.r.output refers to the server output data port1018, also labeled as the r event port on the server, and the a inserving.a.input refers to server input data port 1016, also labeled asthe a port on the server (see FIG. 10).

[0240] Together, these predicates indicate that (1) it is an error forserver 1010 to be in serving mode if client 1028 is not blocked; (2)after server 1010 enters serving mode, a response message is sent orelse an error occurs; and (3) server 1010 receiving a message means thatserver 1010 must enter serving mode. Relationships between control stateand data paths must also be considered, such as:

(client.a

client.blocked)

[0241] In other words, client 1028 must be in blocked mode whenever itsends an argument message.

[0242] The first predicate takes the same form as a constraint; however,since dedicated RPC coordinator 1000 only imports the client:blocked andserver:serving modes (i.e., through RPC client imported state port 1022and RPC server imported state port 1004 respectively), dedicated RPCcoordinator 1000 is not allowed to alter these values to comply. Infact, none of these predicates is explicitly enforced by a runtimesystem. However, the last two can be used as requirements and guaranteesfor interface type-checking.

[0243] 7. System-Level Execution

[0244] Coordination-centric design methodology lets systemspecifications be executed directly, according to the semanticsdescribed above. When components and coordinators are composed intohigher-order structures, however, it becomes essential to considerhazards that can affect system behavior. Examples include conflictingconstraints, in which local resolution semantics may either leave thesystem in an inconsistent state or make it cycle forever, andconflicting actions that undo one another's behavior. In the remainderof this section, the effect of composition issues on system-levelexecutions is explained.

[0245] A. System Control Configurations

[0246] A configuration is the combined control state of asystem—basically, the set of active modes at a point in time. In otherwords, a configuration in coordination-centric design is a bit vectorcontaining one bit for each mode in the system. The bit representing acontrol state is true when the control state is active and false whenthe control state is inactive. Configurations representing the completesystem control state facilitate reasoning on system properties andenable several forms of static analysis of system behavior.

[0247] B. Action-Trigger Propagation

[0248] Triggers are formal parameters for events. As mentioned earlier,there are two types of triggers: (1) control triggers, invoked bycontrol events such as mode change requests, and (2) data flow triggers,invoked by data events such as message arrivals or departures.Components and coordinators can both request mode changes (on the modesvisible to them) and generate new messages (on the message ports visibleto them). Using actions, these events can be propagated through thecomponents and coordinators in the system, causing a cascade of datatransmissions and mode change requests, some of which can cancel otherrequests. When the requests, and secondary requests implied by them, areall propagated through the system, any requests that have not beencanceled are confirmed and made part of the system's new configuration.

[0249] Triggers can be immediately propagated through their respectiveactions or delayed by a scheduling step. Recall that component actionscan be either transparent or opaque. Transparent actions typicallypropagate their triggers immediately, although it is not absolutelynecessary that they do so. Opaque actions typically must always delaypropagation.

[0250] 1. Immediate Propagation

[0251] Some triggers must be immediately propagated through actions, butonly on certain types of transparent actions. Immediate propagation canoften involve static precomputation of the effect of changes, whichmeans that certain actions may never actually be performed. For example,consider a system with a coordinator that has an action that activatesmode A and a coordinator with an action that deactivates mode B wheneverA is activated. Static analysis can be used to determine in advance thatany event that activates A will also deactivate B; therefore, thiseffect can be executed immediately without actually propagating itthrough A.

[0252] 2. Delayed Propagation

[0253] Trigger propagation through opaque actions must typically bedelayed, since the system cannot look into opaque actions to precomputetheir results. Propagation may be delayed for other reasons, such assystem efficiency. For example, immediate propagation requires tightsynchronization among software components. If functionality is spreadamong a number of architectural components, immediate propagation isimpractical.

[0254] C. A Protocol Implemented with a Compound Coordinator

[0255] Multiple coordinators are typically needed in the design of asystem. The multiple coordinators can be used together for a single,unified behavior. Unfortunately, one coordinator may interfere withanother's behavior.

[0256]FIG. 11 shows a combined coordinator 1100 with both preemption andround-robin coordination for controlling access to a resource, asdiscussed above. With reference to FIG. 11, components 1102, 1104, 1106,1108, and 1110 primarily use round-robin coordination, and each includesa component coordination interface 1112, which has a componentarbitrated control port 1114 and a component output message port 1116.However, when a preemptor component 1120 needs the resource, preemptorcomponent 1120 is allowed to grab the resource immediately. Preemptorcomponent 1120 has a preemptor component coordination interface 1122.Preemptor component coordination interface 1122 has a preemptorarbitrated state port 1124, a preemptor output message port 1126, and apreemptor input message port 1128.

[0257] All component coordination interfaces 1112 and preemptorcomponent coordination interface 1122 are connected to a complementarycombined coordinator coordination interface 1130, which has acoordinator arbitrated state port 1132, a coordinator input message port1134, and a coordinator output message port 1136. Combined coordinator1100 is a hierarchical coordinator and internally has a round-robincoordinator (not shown) and a preemption coordinator (not shown).Combined coordinator coordination interface 1130 is connected to acoordination interface to round-robin 1138 and a coordination interfaceto preempt 1140. Coordinator arbitrated state port 1132 is bound to botha token arbitrated control port 1142, which is part of coordinationinterface to round-robin 1138, and to a preempt arbitrated control port1144, which is part of coordination interface to preempt 1140.Coordinator input message port 1134 is bound to an interface to around-robin output message port 1146, and coordinator output messageport 1136 is bound to an interface to round-robin input message port1148.

[0258] Thus preemption interferes with the normal round-robin orderingof access to the resource. After a preemption-based access, the resourcemoves to the component that in round-robin-ordered access would be thesuccessor to preemptor component 1120. If the resource is preempted toofrequently, some components may starve.

[0259] D. Mixing Control and Data in Coordinators

[0260] Since triggers can be control-based, data-based, or both, andactions can produce both control and data events, control and dataflowaspects of a system are coupled through actions. Through combinations ofactions, designers can effectively employ modal data flow, in whichrelative schedules are switched on and off based on the systemconfiguration.

[0261] Relative scheduling is a form of coordination. Recognizing thisand understanding how it affects a design can allow a powerful class ofoptimizations. Many data-centric systems (or subsystems) use conjunctivefiring, which means that a component buffers messages until a firingrule is matched. When matching occurs, the component fires, consumingthe messages in its buffer that caused it to fire and generating amessage or messages of its own. Synchronous data flow systems are thosein which all components have only firing rules with constant messageconsumption and generation.

[0262]FIG. 12A shows a system in which a component N1 1200 is connectedto a component N3 1202 by a data transfer coordinator 1204 and acomponent N2 1206 is connected to component N3 1202 by a second datatransfer coordinator 1208. Component N3 1202 fires when it accumulatesthree messages on a port c 1210 and two messages on a port d 1212. Onfiring, component N3 1202 produces two messages on a port o 1214.Coordination control state tracks the logical buffer depth for thesecomponents. This is shown with numbers representing the logical queuedepth of each port in FIG. 12.

[0263]FIG. 12B shows the system of FIG. 12A in which data transfercoordinator 1204 and second data transfer coordinator 1208 have beenmerged to form a merged data transfer coordinator 1216. Merging thecoordinators in this example provides an efficient static schedule forcomponent firing. Merged data transfer coordinator 1216 fires componentN1 1200 three times and component N2 1206 twice. Merged data transfercoordinator 1216 then fires component N3 1202 twice (to consume allmessages produced by component N1 1200 and component N2 1206).

[0264] Message rates can vary based on mode. For example, a componentmay consume two messages each time it fires in one mode and four eachtime it fires in a second mode. For a component like this, it is oftenpossible to merge schedules on a configuration basis, in which eachconfiguration has static consumption and production rates for allaffected components.

[0265] E. Coordination Transformations

[0266] In specifying complete systems, designers must often specify notonly the coordination between two objects, but also the intermediatemechanism they must use to implement this coordination. While thisintermediate mechanism can be as simple as shared memory, it can also beanother coordinator; hence coordination may be, and often is, layered.For example, RPC coordination often sits on top of a TCP/IP stack or onan IrDA stack, in which each layer coordinates with peer layers on otherprocessing elements using unique coordination protocols. Here, eachlayer provides certain capabilities to the layer directly above it, andthe upper layer must be implemented in terms of them.

[0267] In many cases, control and communication synthesis can beemployed to automatically transform user-specified coordination to aselected set of standard protocols. Designers may have to manuallyproduce transformations for nonstandard protocols.

[0268] F. Dynamic Behavior with Compound Coordinators

[0269] Even in statically bound systems, components may need to interactin a fashion that appears dynamic. For example, RPC-style coordinationoften has multiple clients for individual servers. Here, there is noapparent connection between client and server until one is forged for atransaction. After the connection is forged, however, the coordinationproceeds in the same fashion as dedicated RPC.

[0270] Our approach to this is to treat the RPC server as a sharedresource, requiring resource allocation protocols to control access.However, none of the resource allocation protocols described thus farwould work efficiently under these circumstances. In the followingsubsections, an appropriate protocol for treating the RPC as a sharedresource will be described and how that protocol should be used as partof a complete multiclient RPC coordination class—one that uses the sameRPC coordination interfaces described earlier—will be discussed.

[0271] 1. First Come/First Serve Protocol (FCFS)

[0272]FIG. 13 illustrates a first come/first serve (FCFS) resourceallocation protocol, which is a protocol that allocates a sharedresource to the requester that has waited longest. With reference toFIG. 13, a FCFS component interface 1300 for this protocol has a requestcontrol port 1302, an access control port 1304 and a component outgoingmessage port 1306. A FCFS coordinator 1308 for this protocol has a setof FCFS interfaces 1310 that are complementary to FCFS componentinterfaces 1300, having a FCFS coordinator request control port 1312, aFCFS coordinator access port 1314, and a FCFS coordinator input messageport 1316. When a component 1318 needs to access a resource 1320, itasserts request control port 1302. When granted access, FCFS coordinator1308 asserts the appropriate FCFS coordinator access port 1314,releasing FCFS coordinator request control port 1312.

[0273] To do this, FCFS coordinator 1308 uses a rendezvous coordinatorand two round-robin coordinators. One round-robin coordinator maintainsa list of empty slots in which a component may be enqueued, and theother round-robin coordinator maintains a list showing the nextcomponent to be granted access. When an FCFS coordinator request controlport 1312 becomes active, FCFS coordinator 1308 begins a rendezvousaccess to a binder action. When activated, this action maps theappropriate component 1318 to a position in the round-robin queues. Aseparate action cycles through one of the queues and selects the nextcomponent to access the server. As much as possible, FCFS coordinator1308 attempts to grant access to resource 1320 to the earliest component1318 having requested resource 1320, with concurrent requests determinedbased on the order in the rendezvous coordinator of the respectivecomponents 1318.

[0274] 2. Multiclient RPC

[0275]FIG. 14 depicts a multiclient RPC coordinator 1400 formed bycombining FCFS coordinator 1308 with dedicated RPC coordinator 1000.With reference to FIG. 14, a set of clients 1402 have a set of clientcoordination interfaces 1030, as shown in FIG. 10. In addition,multiclient RPC coordinator 1400 has a set of RPC client coordinationinterfaces 1020, as shown in FIG. 10. For each RPC client coordinationinterface 1020, RPC client input message port 1024, of RPC clientcoordination interface 1020, is bound to the component outgoing messageport 1306 of FCFS coordinator 1308. Message transfer action 1403 servesto transfer messages between RPC client input message port 1024 andcomponent outgoing message port 1306. For coordinating the actions ofmultiple clients 1402, multiclient RPC coordinator 1400 must negotiateaccesses to a server 1404 and keep track of the values returned byserver 1404.

[0276] G. Monitor Modes and Continuations

[0277] Features such as blocking behavior and exceptions can beimplemented in the coordination-centric design methodology with the aidof monitor modes. Monitor modes are modes that exclude all but aselected set of actions called continuations, which are actions thatcontinue a behavior started by another action.

[0278] 1. Blocking Behavior

[0279] With blocking behavior, one action releases control whileentering a monitor mode, and a continuation resumes execution after theanticipated response event. Monitor mode entry must be immediate (atleast locally), so that no unexpected actions can execute before theyare blocked by such a mode.

[0280] Each monitor mode has a list of actions that cannot be executedwhen it is entered. The allowed (unlisted) actions are either irrelevantor are continuations of the action that caused entry into this mode.There are other conditions, as well. This mode requires an exceptionaction if forced to exit. However, this exception action is not executedif the monitor mode is turned off locally.

[0281] When components are distributed over a number of processingelements, it is not practical to assume complete synchronization of thecontrol state. In fact, there are a number of synchronization optionsavailable as detailed in Chou, P “Control Composition and Synthesis ofDistributed Real-Time Embedded Systems”, Ph.D. dissertation, Universityof Washington, 1998.

[0282] 2. Exception Handling

[0283] Exception actions are a type of continuation. When in a monitormode, exception actions respond to unexpected events or events thatsignal error conditions. For example, multiclient RPC coordinator 1400can bind

client.blocked to a monitor mode and set an exception action on +server.serving. This will signal an error whenever the server begins to workwhen the client is not blocked for a response.

[0284] 8. A Complete System Example

[0285]FIG. 15 depicts a large-scale example system under thecoordination-centric design methodology. With reference to FIG. 15, thelarge scale system is a bimodal digital cellular network 1500. Network1500 is for the most part a simplified version of a GSM (global systemfor mobile communications) cellular network. This example shows ingreater detail how the parts of coordination-centric design worktogether and demonstrates a practical application of the methodology.Network 1500 has two different types of cells, a surface cell 1502 (alsoreferred to as a base station 1502) and a satellite cell 1504. Thesecells are not only differentiated by physical position, but by thetechnologies they use to share network 1500. Satellite cells 1504 use acode division multiple access (CDMA) technology, and surface cells 1502use a time division multiple access (TDMA) technology. Typically, thereare seven frequency bands reserved for TDMA and one band reserved forCDMA. The goal is for as much communication as possible to be conductedthrough the smaller TDMA cells, here surface cells 1502, because powerrequirements for a CDMA cells, here satellite cell 1504, increase withthe number of users in the CDMA cell. Mobile units 1506, or wirelessdevices, can move between surface cells 1502, requiring horizontalhandoffs between surface cells 1502. Several surface cells 1502 aretypically connected to a switching center 1508. Switching center 1508 istypically connected to a telephone network or the Internet 1512. Inaddition to handoffs between surface cells 1502, the network must beable to hand off between switching centers 1508. When mobile units 1506leave the TDMA region, they remain covered by satellite cells 1504 viavertical handoffs between cells. Since vertical handoffs requirechanging protocols as well as changing base stations and switchingcenters, they can be complicated in terms of control.

[0286] Numerous embedded systems comprise the overall system. Forexample, switching center 1508 and base stations, surface cells 1502,are required as part of the network infrastructure, but cellular phones,handheld Web browsers, and other mobile units 1506 may be supported foraccess through network 1500. This section concentrates on the softwaresystems for two particular mobile units 1506: a simple digital cellularphone (shown in FIG. 16) and a handheld Web browser (shown in FIG. 24).These examples require a wide variety of coordinators and reusablecomponents. Layered coordination is a feature in each system, because afunction of many subsystems is to perform a layered protocol.Furthermore, this example displays how the hierarchically constructedcomponents can be applied in a realistic system to help manage thecomplexity of the overall design.

[0287] To begin this discussion, we describe the cellular phone indetail, focusing on its functional components and the formalization oftheir interaction protocols. We then discuss the handheld Web browser inless detail but highlight the main ways in which its functionality andcoordination differ from those of the cellular phone. In describing thecellular phone, we use a top-down approach to show how a coherent systemorganization is preserved, even at a high level. In describing thehandheld Web browser, we use a bottom-up approach to illustratecomponent reuse and bottom-up design.

[0288] A. Cellular Phone

[0289]FIG. 16 shows a top-level coordination diagram of the behavior ofa cell phone 1600. Rather than using a single coordinator thatintegrates the components under a single protocol, we use severalcoordinators in concert. Interactions between coordinators occur mainlywithin the components to which they connect.

[0290] With reference to FIG. 16, cell phone 1600 supports digitalencoding of voice streams. Before it can be used, it must beauthenticated with a home master switching center (not shown). Thisauthentication occurs through a registered master switch for each phoneand an authentication number from the phone itself. There are variousauthentication statuses, such as full access, grey-listed, orblacklisted. For cell phone 1600, real-time performance is moreimportant than reliability. A dropped packet is not retransmitted, and alate packet is dropped since its omission degrades the signal less thanits late incorporation.

[0291] Each component of cell phone 1600 is hierarchical. A GUI 1602lets users enter phone numbers while displaying them and query anaddress book 1604 and a logs component 1606. Address book 1604 is adatabase that can map names to phone numbers and vice versa. GUI 1602uses address book 1604 to help identify callers and to look up phonenumbers to be dialed. Logs 1606 track both incoming and outgoing callsas they are dialed. A voice component 1608 digitally encodes anddecodes, and compresses and decompresses, an audio signal. A connectioncomponent 1610 multiplexes, transmits, receives, and demultiplexes theradio signal and separates out the voice stream and calleridentification information.

[0292] Coordination among the above components makes use of several ofthe coordinators discussed above. Between connection component 1610 anda clock 1612, and between logs 1606 and connection component 1610, areunidirectional data transfer coordinators 600 as described withreference to FIG. 6A. Between voice component 1608 and connectioncomponent 1610, and between GUI 1602 and connection component 1610, arebidirectional data transfer coordinators 604, as described withreference to FIG. 6B. Between clock 1612 and GUI 1602 is a stateunification coordinator 606, as described with reference to FIG. 6C.Between GUI 1602 and address book 1604 is a dedicated RPC coordinator1000 as described with reference to FIG. 10, in which address book 1604has client 1028 and GUI 1602 has server 1010.

[0293] There is also a custom GUI/log coordinator 1614 between logs 1606and GUI 1602. GUI/log coordinator 1614 lets GUI 1602 transfer new loggedinformation through an r output message port 1616 on a GUI coordinationinterface 1618 to an r input message port 1620 on a log coordinationinterface 1622. GUI/log coordinator 1614 also lets GUI 1602 choosecurrent log entries through a pair of c output message ports 1624 on GUIcoordination interface 1618 and a pair of c input message ports 1626 onlog coordination interface 1622. Logs 1606 continuously display oneentry each for incoming and outgoing calls.

[0294] 1. GUI Component

[0295]FIG. 17A is a detailed view of GUI component 1602, of FIG. 16.With reference to FIG. 17A, GUI component 1602 has two inner components,a keypad 1700 and a text-based liquid crystal display 1702, as well asseveral functions of its own (not shown). Each time a key press occurs,it triggers an action that interprets the press, depending on the modeof the system. Numeric presses enter values into a shared dialingbuffer. When a complete number is entered, the contents of this bufferare used to establish a new connection through connection component1610. Table 5 shows the action triples for GUI 1602. Mode Trigger ActionIdle

numBuffer.append(keypress.val) Send radio.send(numBuffer.val) +outgoingCall Disconnect Nil Leftarrow AddressBook.forward() + lookupModeRightarrow log.lastcall() + outlog LookupMode LeftarrowAddressBook.forward() Rightarrow AddressBook.backward()

[0296] An “Addr Coord” coordinator 1704 includes an address book mode(not shown) in which arrow key presses are transformed into RPC calls.

[0297] 2. Logs Component

[0298]FIG. 17B is a detailed view of logs component 1606, which tracksall incoming and outgoing calls. With reference to FIG. 17B, both GUIcomponent 1602 and connection component 1610 must communicate with logscomponent 1606 through specific message ports. Those specific messageports include a transmitted number message port 1720, a received numbermessage port 1722, a change current received message port 1724, a changecurrent transmitted message port 1726, and two state ports 1728 and 1729for presenting the current received and current transmitted values,respectively.

[0299] Logs component 1606 contains two identical single-log components:a send log 1730 for outgoing calls and a receive log 1740 for incomingcalls. The interface of logs component 1606 is connected to theindividual log components by a pair of adapter coordinators, Adap1 1750and Adap2 1752. Adap1 1750 has an adapter receive interface 1754, whichhas a receive imported state port 1756 and a receive output message port1758. Adap1 1750 further has an adapter send interface 1760, which has asend imported state port 1762 and a send output message port 1764.Within Adap1, state port 1728 is bound to receive imported state port1756, change current received message port 1724 is bound to receiveoutput message port 1758, received number message port 1722 is bound toa received interface output message port 1766 on a received numbercoordination interface 1768, change current transmitted message port1726 is bound to send output message port 1764, and state port 1729 isbound to Up.rc is bound to send imported state port 1762 .

[0300] 3. Voice Component

[0301]FIG. 18A is a detailed view of voice component 1608 of FIG. 16.Voice component 1608 has a compression component 1800 for compressingdigitized voice signals before transmission, a decompression component1802 for decompressing received digitized voice signals, and interfaces1804 and 1806 to analog transducers (not shown) for digitizing sound tobe transmitted and for converting received transmissions into sound.Voice component 1608 is a pure data flow component containing soundgenerator 1808 which functions as a white-noise generator, a ring tonegenerator, and which has a separate port for each on sound generatorinterface 1810, and voice compression functionality in the form ofcompression component 1800 and decompression component 1802.

[0302] 4. Connection Component

[0303]FIG. 18B is a detailed view of connection component 1610 of FIG.16. With reference to FIG. 18B, connection component 1610 coordinateswith voice component 1608, logs component 1606, clock 1612, and GUI1602. In addition, connection component 1610 is responsible forcoordinating the behavior of cell phone 1600 with a base station thatowns the surface cell 1502 (shown in FIG. 15), a switching center 1508(shown in FIG. 15), and all other phones (not shown) within surface cell1502. Connection component 1610 must authenticate users, establishconnections, and perform handoffs as needed—including appropriatechanges in any low-level protocols (such as a switch from TDMA to CDMA).

[0304]FIG. 19 depicts a set of communication layers between connectioncomponent 1610 of cell phone 1600 and base station 1502 or switchingcenter 1508. With reference to FIG. 19, has several subcomponents, orlower-level components, each of which coordinates with an equivalent, orpeer, layer on either base station 1502 or switching center 1508. Thesubcomponents of connection component 1610 include a cell phone callmanager 1900, a cell phone mobility manager 1902, a cell phone radioresource manager 1904, a cell phone link protocol manager 1906, and acell phone transport manager 1908 which is responsible for coordinatingaccess to and transferring data through the shared airwaves TDMA andCDMA coordination. Each subcomponent will be described in detailincluding how each fits into the complete system.

[0305] Base station 1502 has a call management coordinator 1910, amobility management coordinator 1912, a radio resource coordinator 1914(BSSMAP 1915), a link protocol coordinator 1916 (SCCO 1917), and atransport coordinator 1918 (MTP 1919). Switching center 1508 has aswitching center call manager 1920, a switching center mobility manager1922, (a BSSMAP 1924, a SCCP 1926, and an MTP 1928.

[0306] a. Call Management

[0307]FIG. 20 is a detailed view of a call management layer 2000consisting of cell phone call manager 1900, which is connected toswitching center call manager 1920 by call management coordinator 1910.With reference to FIG. 20, call management layer 2000 coordinates theconnection between cell phone 1600 and switching center 1508. Callmanagement layer 2000 is responsible for dialing, paging, and talking.Call management layer 2000 is always present in cell phone 1600, thoughnot necessarily in Internet appliances (discussed later). Cell phonecall manager 1900 includes a set of modes (not shown) for callmanagement coordination that consists of the following modes:

[0308] Standby

[0309] Dialing

[0310] RingingRemote

[0311] Ringing

[0312] CallInProgress

[0313] Cell phone call manager 1900 has a cell phone call managerinterface 2002. Cell phone call manager interface 2002 has a portcorresponding to each of the above modes. The standby mode is bound to astandby exported state port 2010. The dialing mode is bound to a dialingexported state port 2012. The RingingRemote mode is bound to aRingingRemote imported state port 2014. The Ringing mode is bound to aringing imported state port 2016. The CallInProgress mode is bound to aCallInProgress arbitrated state port 2018.

[0314] Switching center call manager 1920 includes the following modes(not shown) for call management coordination at the switching center:

[0315] Dialing

[0316] RingingRemote

[0317] Paging

[0318] CallInProgress

[0319] Switching center call manager 1920 has a switching center callmanager coordination interface 2040, which includes a port for each ofthe above modes within switching center call manager 1920.

[0320] When cell phone 1600 requests a connection, switching center 1508creates a new switching center call manager and establishes a callmanagement coordinator 1910 between cell phone 1600 and switching centercall manager 1920.

[0321] b. Mobility Management

[0322] A mobility management layer authenticates mobile unit 1506 orcell phone 1600. When there is a surface cell 1502 available, mobilitymanager 1902 contacts the switching center 1508 for surface cell 1502and transfers a mobile unit identifier (not shown) for mobile unit 1506to switching center 1508. Switching center 1508 then looks up a homemotor switching center for mobile unit 1506 and establishes a set ofpermissions assigned to mobile unit 1506. This layer also acts as aconduit for the call management layer. In addition, the mobilitymanagement layer performs handoffs between base stations 1502 andswitching centers 1508 based on information received from the radioresource layer.

[0323] C. Radio Resource

[0324] In the radio resource layer, radio resource manager 1904, choosesthe target base station 1502 and tracks changes in frequencies, timeslices, and CDMA codes. Cell phones may negotiate with up to 16 basestations simultaneously. This layer also identifies when handoffs arenecessary.

[0325] d. Link Protocol

[0326] The link layer manages a connection between cell phone 1600 andbase station 1502. In this layer, link protocol manager 1906 packagesdata for transfer to base station 1502 from cell phone 1600.

[0327] e. Transport

[0328]FIG. 21A is a detailed view of transport component 1908 ofconnection component 1610. Transport component 1908 has twosubcomponents, a receive component 2100 for receiving data and atransmit component 2102 for transmitting data. Each of thesesubcomponents has two parallel data paths a CDMA path 2104 and aTDMA/FDMA path 2106 for communicating in the respective networkprotocols.

[0329]FIG. 21B is a detailed view of a CDMA modulator 2150, whichimplements a synchronous data flow data path. CDMA modulator 2150 takesthe dot-product of an incoming data signal along path 2152 and a storedmodulation code for cell phone 1600 along path 2154, which is a sequenceof chips, which are measured time signals having a value of −1 or +1.

[0330] Transport component 1908 uses CDMA and TDMA technologies tocoordinate access to a resource shared among several cell phones 1600,i.e., the airwaves. Transport components 1908 supersede the FDMAtechnologies (e.g., AM and FM) used for analog cellular phones and forradio and television broadcasts. In FDMA, a signal is encoded fortransmission by modulating it with a carrier frequency. A signal isdecoded by demodulation after being passed through a band pass filter toremove other carrier frequencies. Each base station 1502 has a set offrequencies—chosen to minimize interference between adjacent cells. (Thearea covered by a cell may be much smaller than the net range of thetransmitters within it.)

[0331] TDMA, on the other hand, coordinates access to the airwavesthrough time slicing. Cell phone 1600 on the network is assigned a smalltime slice, during which it has exclusive access to the media. Outsideof the small time slice, cell phone 1600 must remain silent. Decoding isperformed by filtering out all signals outside of the small time slice.The control for this access must be distributed. As such, each componentinvolved must be synchronized to observe the start and end of the smalltime slice at the same instant.

[0332] Most TDMA systems also employ FDMA, so that instead of sharing asingle frequency channel, cell phones 1600 share several channels. Theband allocated to TDMA is broken into frequency channels, each with acarrier frequency and a reasonable separation between channels. Thususer channels for the most common implementations of TDMA can berepresented as a two-dimensional array, in which the rows representfrequency channels and the columns represent time slices.

[0333] CDMA is based on vector arithmetic. In a sense, CDMA performsinter-cell-phone coordination using data flow. Instead of breaking upthe band into frequency channels and time slicing these, CDMA regardsthe entire band as an n-dimensional vector space. Each channel is a codethat represents a basis vector in this space. Bits in the signal arerepresented as either 1 or −1, and the modulation is the inner productof this signal and a basis vector of mobile unit 1506 or cell phone1600. This process is called spreading, since it effectively takes anarrowband signal and converts it into a broadband signal.

[0334] Demultiplexing is simply a matter of taking the dot-product ofthe received signal with the appropriate basis vector, obtaining theoriginal 1 or −1. With fast computation and the appropriate codes orbasis vectors, the signal can be modulated without a carrier frequency.If this is not the case, a carrier and analog techniques can be used tofill in where computation fails. If a carrier is used, however, allunits use the same carrier in all cells.

[0335]FIG. 22 shows TDMA and CDMA signals for four cell phones 1600.With reference to FIG. 22, for TDMA, each cell phone 1600 is assigned atime slice during which it can transmit. Cell phone 1 is assigned timeslice t0, cell phone 2 is assigned time slice t1, cell phone 3 isassigned time slice t2, and cell phone 4 is assigned time slice t3. ForCDMA, each cell phone 1600 is assigned a basis vector that it multiplieswith its signal. Cell phone 1 is assigned the vector: $\begin{pmatrix}{- 1} \\1 \\{- 1} \\1\end{pmatrix}\quad$

[0336] Cell phone 2 is assigned the vector: $\begin{pmatrix}1 \\{- 1} \\1 \\{- 1}\end{pmatrix}\quad$

[0337] Cell phone 3 is assigned the vector: $\begin{pmatrix}1 \\1 \\{- 1} \\{- 1}\end{pmatrix}\quad$

[0338] Cell phone 4 is assigned the vector: $\begin{pmatrix}{- 1} \\{- 1} \\1 \\1\end{pmatrix}\quad$

[0339] Notice that these vectors form an orthogonal basis.

[0340] B. Handheld Web Browser

[0341] In the previous subsection, we demonstrated our methodology on acell phone with a top-down design approach. In this subsection, wedemonstrate our methodology with a bottom-up approach in building ahandheld Web browser.

[0342]FIG. 23A is a LCD touch screen component 2300 for a Web browserGUI (shown in FIG. 24A) for a wireless device 1506. With reference toFIG. 23A, a LCD touch screen component 2300, has an LCD screen 2302 anda touch pad 2304.

[0343]FIG. 23B is a Web page access component 2350 for fetching andformatting web pages. With reference to FIG. 23B, web access component2350 has a page fetch subcomponent 2352 and a page format subcomponent2354. Web access component 2350 reads hypertext markup language (HTML)from a connection interface 2356, sends word placement requests to adisplay interface 2358, and sends image requests to the connectioninterface 2356. Web access component 2350 also has a character inputinterface to allow users to enter page requests directly and to fill outforms on pages that have forms.

[0344]FIG. 24A shows a completed handheld Web browser GUI 2400. Withreference to FIG. 24A, handheld Web browser GUI 2400, has LCD touchscreen component 2300, web access component 2350, and a pen strokerecognition component 2402 that translates pen strokes entered on touchpad 2304 into characters.

[0345]FIG. 24B shows the complete component view of a handheld Webbrowser 2450. With reference to FIG. 24B, handheld Web browser 2450 isformed by connecting handheld Web browser GUI 2400 to connectioncomponent 1610 of cell phone 1600 (described with reference to FIG. 16)with bi-directional data transfer coordinator 604 (described withreference to FIG. 6B). Handheld Web browser 2450 is an example of mobileunit 1506, and connects to the Internet through the cellularinfrastructure described above. However, handheld Web browser 2450 hasdifferent access requirements than does cell phone 1600. For handheldWeb browser 2450, reliability is more important than real-time delivery.Dropped packets usually require retransmission, so it is better todeliver a packet late than to drop it. Real-time issues primarily affectdownload time and are therefore secondary. Despite this, handheld Webbrowser 2450 must coordinate media access with cell phones 1600, and soit must use the same protocol as cell phones 1600 to connect to thenetwork. For that reason, handheld Web browser 2450 can reuse connectioncomponent 1610 from cell phone 1600.

[0346] Debugging Techniques

[0347] In concept, debugging is a simple process. A designer locates thecause of undesired behavior in a system and fixes the cause. Inpractice, debugging—even of sequential software—remains difficult.Embedded systems are considerably more complicated to debug thansequential software, due to factors such as concurrence, distributedarchitectures, and real-time concerns. Issues taken for granted insequential software, like a schedule that determines the order of allevents (the program), are nonexistent in a typical distributed system.Locating and fixing bugs in these complex systems requires many factors,including an understanding of the thought processes underpinning thedesign.

[0348] Prior art research into debugging distributed systems is diverseand eclectic and lacks any standard notations. This application uses astandardized notation both to describe the prior art and the presentinvention. As a result of this standardized notation, the principles inthe prior art follow those published in the referenced works. However,the specific notation, theorems, etc., may differ.

[0349] The two general classes of debugging techniques are event-baseddebugging and state-based debugging. Most debugging techniques forgeneral-purpose distributed systems are event based. Event-baseddebugging techniques operate by collecting event traces from individualsystem components and causally relating those event traces. Thesetechniques require an ability to determine efficiently the causalordering among any given pair of events. Determining the causal ordercan be difficult and costly.

[0350] Events may be primitive, or they may be hierarchical clusters ofother events. Primitive events are abstractions of individual localoccurrences that might be important to a debugger. Examples of primitiveevents in sequential programs are variable assignments and subroutineentries or returns. Primitive events for distributed systems includemessage send and receive events.

[0351] State-based debugging techniques are less commonly used indebugging distributed systems. State-based debugging techniquestypically operate by presenting designers with views or snapshots of aprocess state. Distributed systems are not tightly synchronized, and sothese techniques traditionally involve only the state of individualprocesses. However, state-based debugging techniques can be applied moregenerally by relaxing the concept of an “instant in time” so that it canbe effectively applied to asynchronous processes.

[0352] 1. Event-Based Debugging

[0353] In this section, prior art systems for finding and trackingmeaningful event orderings, despite limits in observation, aredescribed. Typical ways in which event orderings are used invisualization tools through automated space/time diagrams are thendescribed.

[0354] A. Event Order Determination and Observation

[0355] The behavior of a software system is determined by the eventsthat occur and the order in which they occur. For sequential systems,this seems almost too trivial to mention; of course, a given set ofevents, such as

{x:=2, x:=x*2, x:=5, y:=x},

[0356] arranged in two different ways may describe two completelydifferent behaviors. However, since a sequential program is essentiallya complete schedule of events, ordering is explicit. Sequentialdebugging tools depend on the invariance of this event schedule to letprogrammers reproduce failures by simply using the same inputs. Indistributed systems, as in any concurrent system, it is neitherpractical nor efficient to completely schedule all events. Concurrentsystems typically must be designed with flexible event ordering.

[0357] Determining the order in which events occur in a distributedsystem is subject to the limits of observation. An observation is anevent record collected by an observer. An observer is an entity thatwatches the progress of an execution and records events but does notinterfere with the system. To determine the order in which two eventsoccur, an observer must measure them both against a common reference.

[0358]FIG. 25 shows a typical space/time diagram 2500, with spacerepresented on a vertical axis 2502 and time represented on a horizontalaxis 2504. With reference to FIG. 25, space/time diagram 2500 provides astarting point for discussing executions in distributed systems.Space/time diagram 2500 gives us a visual representation for discussingevent ordering and for comparing various styles of observation. A set ofhorizontal world lines 2506, 2508, and 2510 each represent an entitythat is stationary in space. The entities represented by horizontalworld lines 2506, 2508, and 2510 are called processes and typicallyrepresent software processes in the subject system. The entities canalso represent any entity that generates events in a sequential fashion.The spatial separation in the diagram, along vertical axis 2502,represents a virtual space, since several processes might execute on thesame physical hardware. A diagonal world line 2512 is called a messageand represents discrete communications that pass between two processes.A sphere 2514 represents an event. In subsequent figures vertical axis2502 and horizontal axis 2504 are omitted from any space/time diagrams,unless vertical axis 2502 and horizontal axis 2504 provide additionalclarity to a particular figure.

[0359]FIG. 26 shows a space/time diagram 2600 of two differentobservations of a single system execution, taken by a first observer2602 and a second observer 2604. With reference to FIG. 26, firstobserver 2602 and second observer 2604 are entities that record eventoccurrence. First observer 2602 and second observer 2604 must eachreceive distinct notifications of each event that occurs and each mustrecord the events in some total order. First observer 2602 and secondobserver 2604 are represented in space/time diagram 2600 as additionalprocesses, or horizontal world lines. Each event recorded requires asignal from its respective process to both first observer 2602 andsecond observer 2604. The signals from an event x 2606 on a process 2608to both first observer 2602 and second observer 2604 are embodied inmessages 2610 and 2612, respectively. First observer 2602 records eventx 2606 as preceding an event y 2614. However, second observer 2604records event y 2614 as preceding event x 2606. Such effects may becaused by nonuniform latencies within the system.

[0360] However, the observations of first observer 2602 and secondobserver 2604 are not equally valid. A valid observation is typically anobservation that preserves the order of events that depend on eachother. Second observer 2604 records the receipt of a message 2616 beforethat message is transmitted. Thus the observation from second observer2604 is not valid.

[0361]FIG. 27 shows a space/time diagram 2700 for a special, idealobserver, called the real-time observer (RTO) 2702. With reference toFIG. 27, RTO 2702 can view each event immediately as it occurs. Due tothe limitations of physical clocks, and efficiency issues in employingthem, it is usually not practical to implement RTO 2702. However, RTO2702 represents an upper bound on precision in event-orderdetermination.

[0362]FIG. 28 shows a space/time graph 2800 showing two validobservations of a system taken by two separate observers: RTO 2702 and athird observer 2802. With reference to FIG. 28, there is nothing specialabout the ordering of the observation taken by RTO 2702. Events d 2804,e 2806, and f 2808 are all independent events in this execution.Therefore, the observation produced by RTO 2702 and the observationproduced by third observer 2802 can each be used to reproduce equivalentexecutions of the system. Any observation in which event dependenciesare preserved is typically equal in value to an observation by RTO 2702.However, real-time distributed systems may need additional processes toemulate timing constraints.

[0363]FIG. 29 is a space/time diagram 2900 of a methodological observer,called the discrete Lamport Observer (DLO) 2902, that records each eventin a set of ordered bins. With reference to FIG. 29, DLO 2902 records anevent 2904 in an ordered bin 2906 based on the following rule: eachevent is recorded in the leftmost bin that follows all events on whichit depends. DLO 2902 views events discretely and does not need a clock.DLO 2902 does, however, require explicit knowledge of event dependency.To determine the bin in which each event must be placed, DLO 2902 needsto know the bins of the immediately preceding events. The observationproduced by DLO 2902 is also referred to as a topological sort of thesystem execution's event graph.

[0364] In the following, E is the set of all events in an execution. Theimmediate predecessor relation,

E×E, includes all pairs (e_(a), e_(b)) such that:

[0365] a) If e_(a) and e_(b) are on the same process, e_(a) precedese_(b) with no intermediate events.

[0366] b) If e_(b) is a receive event, e_(a) is the send event thatgenerated the message. Given these conditions, e_(a) is called theimmediate predecessor of e_(b).

[0367] Each event has at most two immediate predecessors. Therefore, DLO2902 need only find the bins of at most two records before eachplacement. The transitive closure of the immediate predecessor relationforms a causal relation. The causal relation,

E×E, is the smallest transitive relation such that e_(i)→e_(j)

e_(j).

[0368] This relation defines a partial order of events and furtherlimits the definition of a valid observation. A valid observation is anordered record of events from a given execution, i.e., (R,

), where eεE

(record(e))εR and

is an ordering operator. A valid observation has:

e_(i); e_(j)εE, e_(i)

e_(j)

record(e_(i))

record(e_(j))

[0369] The dual of the causal relation is a concurrence relation. Theconcurrence relation, E×E, includes all pairs (e_(a), e_(b)) such thatneither e_(a)

e_(b) nor e_(b)

e_(a). While the causal relation is transitive, the concurrence relationis not. The concurrence relation is symmetric, while the causal relationis not.

[0370] B. Event-Order Tracking

[0371] Debugging typically requires an understanding of the order inwhich events occur. Above, observers were presented as separateprocesses. While that treatment simplified the discussion of observersit is typically not a practical implementation of an observer. When theobserver is implemented as a physical process, the signals to indicateevents would have to be transformed into physical messages and thesystem would have to be synchronized to enable all messages to arrive ina valid order.

[0372]FIG. 30 depicts a space/time graph 3000 with each event having alabel 3002. With reference to FIG. 30, DLO 2902 can accurately placeevent records in their proper bins—even if received out of order—as longas it knows the bins of the immediate predecessors. If we know the binsin which events are recorded, we can determine something about theircausality. Fortunately, it is easy to label each event with the numberof its intended bin. Labels 3002 are analogous to time and are typicallycalled Lamport timestamps.

[0373] A Lamport timestamp is an integer t associated with an evente_(i) such that

e_(i)

e_(j)

t(e_(i))<t(e_(j))

[0374] Lamport timestamps can be assigned as needed, provided the labelsof an event's immediate predecessors are known. This information can bemaintained with a local counter, called a Lamport clock (not shown),t_(Pi), on each process, P_(i). The clock's value is transmitted witheach message M_(j) as t_(Mj). Clock value t_(Pt) is updated with eachevent, as follows: ${tpi} = \begin{Bmatrix}{{{\max \left( {{tMj},{tpi}} \right)} + 1};} & {{if}\quad e\quad {is}\quad a\quad {receive}\quad {event}} \\{{{tpi} + 1};} & {otherwise}\end{Bmatrix}$

[0375] A labeling mechanism is said to characterize the causal relationif, based on their labels alone, it can be determined whether two eventsare causal or concurrent. Although Lamport timestamps are consistentwith causality (if t(e_(i))≧t(e_(i)), then e_(i)

e_(j)), they do not characterize the causal relation.

[0376]FIG. 31 is a space/time graph 3100 that demonstrates the inabilityof scalar timestamps to characterize causality between events. Withreference to FIG. 31, space/time graph 3100 shows event e₁ 3102, evente₂ 3104, and event e₃ 3106. e₁ 3102 causes e₂ 3104, and also e₁ 3102 isconcurrent with e₃ 3106 e₂ 3104 is concurrent with e₃ 3106 and it can beshown that e₃ 3106 appears, when scalar timestamps are used, concurrentwith both e₁ 3102 and e₂ 3104. However, since e₁ 3102

e₂ 3104 it is not possible for e₃ 3106 to be concurrent with both.

[0377] Event causality can be tracked completely using explicit eventdependence graphs, with directed edges from each event to its immediatepredecessors. Unfortunately, this method cannot store enough informationwith each record to determine whether two arbitrarily chosen events arecausally related without traversing the dependence graph.

[0378] Other labeling techniques, such as vector timestamps, cancharacterize causality. The typical formulation of vector timestamps isbased on the cardinality of event histories. A basis for vectortimestamp is established based on the following definitions andtheorems. An event history, H(e_(j)), of an event e_(j) is the set ofall events, e_(i), such that either since e_(i)

e_(j) or e₁

e_(i)=e_(j). The event history can be projected against specificprocesses. For a process P_(j): the P_(j) history projection ofH(e_(i)), H_(Pj) (e_(i)), is the intersection of H(e_(i)) and the set ofevents local to P_(j). The event graph represented by a space/timediagram can be partitioned into equivalence classes, with one class foreach process. The set of events local to P_(j) is just the P_(j)equivalence class.

[0379] The intersection of any two projections from the same process isidentical to at least one of the two projections. Two historyprojections from a single process, Hp(a) and Hp(b), must satisfy one ofthe following:

[0380] a) Hp(a)⊂Hp(b)

[0381] b) Hp(a)=Hp(b)

[0382] c) Hp(a)⊃Hp(b)

[0383] The cardinality of H_(Pj) (e_(i)) is thus the number of eventslocal to P_(j) that causally precede e_(i) and e_(i) itself. Since localevents always occur in sequence, we can uniquely identify an event byits process and the cardinality of its local history.

[0384] For events e_(a); e_(b) with e_(a)≠e_(b), H_(Pea)(e_(a))

H_(Pea) (e_(b))

e_(a)

e_(b)

[0385]FIG. 32 shows a space/time diagram 3200 with vector timestampedevents. A vector timestamp 3202 is a vector label, t_(e), assigned toeach event, eεE, such that the i^(th) element represents [H_(Pi)(e)].Given two events, e₁ and e₂, we can determine their causal ordering: ifvector t_(ei) has a smaller value for its own process's entry than theother, t_(ej), has at that same position, then ei

ej. If both vectors have larger values for their own process entries,then e_(i)∥e_(j). It is not possible for both events to have smallervalues for their own entries because for events e_(a) and e_(b), e_(a)

e_(b) implies H_(Pea) (e_(a))

H_(Pea) (e_(b)). It is not necessary to know the local processes ofevents to determine their causal order using vector timestamps.

[0386] The causal order of two vector timestamped events, e_(a) ande_(b), from unknown processes can be determined with anelement-by-element comparison of their vector timestamps:$\left. {\underset{i = 1}{\overset{n}{}}{{t_{ea}\lbrack i\rbrack} \leq {t_{ea}\lbrack i\rbrack}}}\Rightarrow\left. e_{a}\rightarrow e_{a} \right. \right.$$\left. {{\underset{i = 1}{\overset{n}{}}{{t_{ea}\lbrack i\rbrack} \leq {t_{eb}\lbrack i\rbrack}}{{\underset{i = 1}{\overset{n}{}}{{t_{eb}\lbrack i\rbrack} \leq {t_{ea}\lbrack i\rbrack}}}}}}\Rightarrow{e_{a}{e_{b}}} \right.$

[0387] Thus vector timestamps both fully characterize causality anduniquely identify each event in an execution.

[0388] Computing vector timestamps at runtime is similar to Lamporttimestamp computation. Each process (P_(s)) contains a vector clock({circumflex over (t)}_(Ps)) with elements for every process in thesystem, where {circumflex over (t)}_(Ps)[s] always equals the number ofevents local to P_(s). Snapshots of this vector counter are used tolabel each event, and snapshots are transmitted with each message. Therecipient of a message with a vector snapshot can update its own vectorcounter ({circumflex over (t)}_(Pr)) by replacing it withsup({circumflex over (t)}_(Ps), {circumflex over (t)}_(Pr)), theelement-wise maximum of {circumflex over (t)}_(Ps) and {circumflex over(t)}_(Pr).

[0389] This technique places enough information with each message todetermine message ordering. It is performed by comparing snapshotsattached to each message. However, transmission of entire snapshots isusually not practical, especially if the system contains a large numberof processes.

[0390] Vector clocks can however be maintained without transmittingcomplete snapshots. A transmitting process, P_(s), can send a list thatincludes only those vector clock values that have changed since its lastmessage. A recipient, P_(r), then compares the change list to itscurrent elements and updates those that are smaller. This requires eachprocess to maintain several vectors: one for itself and one for eachprocess to which it has sent messages. However, change lists do notcontain enough information to independently track message order.

[0391] The expense of maintaining vector clocks can be a strongdeterrent to employing them. Unfortunately, no technique with smallerlabels can characterize causality. It has been proven that the dimensionof the causal relation for an N-process distributed execution is N, andhence N-element vectors are the smallest labels characterizingcausality.

[0392] The problem results from concurrence, without which Lamport timewould be sufficient. Concurrence can be tracked with concurrency maps,where each event keeps track of all events with which it is concurrent.Since the maps characterize concurrency, adding Lamport time lets themalso characterize causality (the concurrency information disambiguatesthe scalar time). Unfortunately, concurrency maps can only beconstructed after-the-fact, since doing so requires an examination ofevents from all processes.

[0393] In some situations, distinguishing between concurrency andcausality is not a necessity, but merely a convenience. There arecompact labeling techniques that allow better concurrence detection thanLamport time. One such technique uses interval clocks in which eachevent record is labeled with its own Lamport time and the Lamport timeof its earliest successor. This label then represents a Lamport timeinterval, during which the corresponding event was the latest known bythe process. This gives each event a wider region with which to detectconcurrence (indicated by overlapping intervals).

[0394] In cases in which there is little or no cross-process causality(few messages), interval timestamps are not much better than Lamporttimestamps. In cases with large numbers of messages, however, intervaltimestamps can yield better results.

[0395] C. Space/Time Displays in Debugging Tools

[0396] Space/time diagrams have typically proven useful in discussingevent causality and concurrence. Space/time diagrams are also oftenemployed as the user display in concurrent program debugging tools.

[0397] The Los Alamos parallel debugging system uses a text basedtime-process display, and Idd uses a graphic display. Both of these,however, rely on an accurate global real-time clock (impractical in mostsystems).

[0398]FIG. 33 shows a partial order event tracer (POET) display 3300.The partial order event tracer (POET) system supports several differentlanguages and run-time environments, including Hermes, a high-levelinterpreted language for distributed systems, and Java. With referenceto FIG. 33, POET display 3300 distinguishes among several types ofevents by shapes, shading, and alignment of corresponding message lines.

[0399] A Distributed Program Debugger (DPD) is based on a RemoteExecution Manager (REM) framework. The REM framework is a set of serverson interconnected Unix machines in which each server is a Unixuser-level process. Processes executing in this framework can create andcommunicate with processes elsewhere in the network as if they were allon the same machine. DPD uses space/time displays for debuggingcommunication only, and it relies on separate source-level debuggers forindividual processes.

[0400] 2. Abstraction in Event-Based Debugging

[0401] Simple space/time displays can be used to present programmerswith a wealth of information about distributed executions. Typically,however, space/time diagrams are too abstract to be an ultimatedebugging solution. Space/time diagrams show high-level events andmessage traffic, but they do not support designer interaction with thesource code. On the other hand, simple space/time diagrams may sometimeshave too much detail. Space/time diagrams display each distinctlow-level message that contributes to a high-level transaction withoutsupport for abstracting the transaction.

[0402]FIG. 34 is a space/time diagram 3400 having a first compound event3402 and a second compound event 3404. With reference to FIG. 34, eventhough a pair of primitive events are either causally related orconcurrent, first and second compound events 3402 and 3404, or any otherpair of compound events, might be neither causally related norconcurrent. Abstraction is typically applied across twodimensions—events and processes—to aid in the task of debuggingdistributed software. Event abstraction represents sequences of eventsas single entities. A group of events may occasionally have a specificsemantic meaning that is difficult to recognize, much as streams ofcharacters can have a meaning that is difficult to interpret withoutproper spacing and punctuation. Event abstraction can in somecircumstances complicate the relationships between events.

[0403] Event abstraction can be applied in one of three ways: filtering,clustering, and interpretation. With event filtering, a programmerdescribes event types that the debugger should ignore, which are thenhidden from view. With clustering, the debugger collects a number ofevents and presents the group as a single event. With interpretation,the debugger parses the event stream for event sequences with specificsemantic meaning and presents them to a programmer.

[0404] Process abstraction is usually applied only as hierarchicalclustering. The remainder of this section discusses these specific eventand process abstraction approaches.

[0405] A. Event Filtering and Clustering

[0406] Event filtering and clustering are techniques used to hide eventsfrom a designer and thereby reduce clutter. Event filters excludeselected events from being tracked in event-based debugging techniques.In most cases, this filtering is implicit and cannot be modified withoutchanging the source code because the source code being debugged isdesigned to report only certain events to the debugger. When deployed,the code will report all such events to the tool. This approach isemployed in both DPD and POET, although some events may be filtered fromthe display at a later time.

[0407] An event cluster is a group of events represented as a singleevent. The placement of an event in a cluster is based on simpleparameters, such as virtual time bounds and process groups. Eventclusters can have causal ambiguities. For example, one cluster maycontain events that causally precede events in a second cluster, whileother events causally follow certain events in the second cluster.

[0408]FIG. 35 shows a POET display 3500 involving a first convex eventcluster 3502 and a second convex event cluster 3504. POET uses avirtual-time-based clustering technique that represents convex eventclusters as single abstract events. A convex event cluster is a set ofevent instances, C, such that for events

a, b, cεE with a, cεC, a

b^ b

c

bεC

[0409] Convex event clusters, unlike generic event clusters, cannotoverlap.

[0410] B. Event Interpretation (Specific Background for BehavioralAbstraction)

[0411] The third technique for applying event abstraction isinterpretation, also referred to as behavioral abstraction. Both termsdescribe techniques that use debugging tools to interpret the behaviorrepresented by sequences of events and present results to a designer.Most approaches to behavioral abstraction let a designer describesequences of events using expressions, and the tools recognize thesequence of events through a combination of customized finite automatafollowed by explicit checks. Typically, matched expressions generate newevents.

[0412] 1. Event Description Language (EDL)

[0413] One of the earliest behavioral abstraction technique was Bates'sevent description language (EDL), in which event streams arepattern-matched using shuffle automata. A match produces a new eventthat can, in turn, be part of another pattern. Essentially, abstractevents are hierarchical and are built from the bottom up.

[0414] This approach can recognize event patterns that containconcurrent events. There are, however, several weaknesses in thisapproach. First, shuffle automata match events from a linear stream,which is subject to a strong observational bias. In addition, even ifthe stream constitutes a valid observation, interleaving may cause falseintermediates between an event and its immediate successor. Finally,concurrent events appear to occur in some specific order.

[0415] Bates partially compensates for these problems in three ways.First, all intermediates between two recognized events areignored—hence, false intermediates are skipped. Unfortunately, trueintermediates are also skipped, making error detection difficult.Second, the shuffle operator, Δ, is used to identify matches withconcurrent events. Unfortunately, shuffle recognizes events that occurin any order, regardless of whether they are truly ordered in thecorresponding execution. For example, e₁Δe₂ can match with either e₁

e₂ or e₂

e₁ in the event stream, but this means the actual matches could be: e₁

e₂, e₂

e₁, in addition to the e₁∥e₂ that the programmer intended to match.Third, the programmer can prescribe explicit checks to be performed oneach match before asserting the results. However, the checks allowed donot include causality or concurrence checks.

[0416] 2. Chain Expressions

[0417] Chain expressions, used in the Ariadne parallel debugger, are analternate way to describe distributed behavior patterns that have bothcausality and concurrence. These behavioral descriptions are based onchains of events (abstract sequences not bound to processes), p-chains(chains bound to processes), and pt-chains (composed p-chains). Thesyntax for describing chain expressions is fairly simple, with <a b >representing two causally related events and |[a b]| representing twoconcurrent events.

[0418] The recognition algorithm has two functions. First, the algorithmrecognizes the appropriate event sequence from a linear stream, using anondeterminate finite automaton (NFA). Second, the algorithm checks therelationships between specific events.

[0419] For example, when looking for sequences that match the expression<|[a b]|c> (viz., a and b are concurrent, and both causally precede c),Ariadne will find the sequence a b c and then verify the relationshipsamong them. Unfortunately, the fact that sequences are picked in orderfrom a linear stream before relationships are checked can cause certainmatches to be missed. For example, |[a b]| and |[b a]| should have thesame meaning, but they do not cause identical matches. This is becauseAriadne uses NFAs as the first stage in event abstraction. In thetotally ordered stream to which an NFA responds, either a will precedeb, preventing the NFA for the second expression from recognizing thestring, or b will precede a, preventing the NFA for the first expressionfrom recognizing the string.

[0420] 3. Distributed Abstraction

[0421] The behavioral abstraction techniques described so far rely oncentralized abstraction facilities. These facilities can be distributed,as well. The BEE (Basis for distributed Event Environments) project is adistributed, hierarchical, event-collection system, with debuggingclients located with each process.

[0422]FIG. 36 show a Basis for distributed Event Environments (BEE)abstraction facility 3600 for a single client. With reference to FIG.36, event interpretation is performed at several levels. The first is anevent sensor 3602, inserted into the source of the program under testand invoked whenever a primitive event occurs during execution. The nextlevel is an event generator 3604, where information—including timestampsand process identifiers—is attached to each event. Event generator 3604uses an event table 3606 to determine whether events should be passed toan event handler 3608 or simply dropped. Event handler 3608 managesevent table 3606 within event generator 3604. Event handler 3608 filtersand collects events and routes them to appropriate event interpreters(not shown). Event interpreters (not shown) gather events from a numberof clients (not shown) and aggregate them for presentation to aprogrammer. Clients and their related event interpreters are placedtogether in groups managed by an event manager (not shown). A weaknessof this technique is that it does not specifically track causality.Instead, this technique relies on the real-timestamps attached tospecific primitive or abstract events. However, as discussed above thesetimestamps are not able to characterize causality.

[0423] C. Process Clustering

[0424] Most distributed computing environments feature flat processstructures, with few formally stated relationships among processes.Automatic process clustering tools can partially reverse-engineer ahierarchical structure to help remove spurious information from adebugger's view. Intuitively, a good cluster hierarchy should reveal, atthe top level, high-level system behavior, and the resolution shouldimprove proportionally with the number of processes exposed. A poorcluster hierarchy would show very little at the top level and wouldrequire a programmer to descend several hierarchical levels beforegetting even a rough idea about system behavior. Process clusteringtools attempt to identify common interaction patterns—such asclient-server, master-slave, complex server, layered system, and soforth. When these patterns are identified, the participants areclustered together. Clusters can then serve as participants ininteraction patterns to be further clustered. These cluster hierarchiesare strictly trees, as shown in FIG. 37, which depicts a hierarchicalconstruction of process clusters 3700. With reference to FIG. 37, asquare node 3702 represents a process (not shown) and a round node 3704represents a process cluster (not shown).

[0425] Programmers can choose a debugging focus, in which they specifythe aspects and detail levels they want to use to observe an execution.With reference to FIG. 37, a representative debugging focus thatincludes nodes I, J, E, F, G, and H is shown. One drawback of thisapproach is that when a parent cluster is in focus, none of its childrencan be. For example, if we wanted to look at process K in detail, wewould also need to expose at least as much detail for processes E and Land process cluster D.

[0426] Each process usually participates in many types of interactionswith other processes. Therefore, the abstraction tools mustheuristically decide between several options. These decisions have asubstantial impact on the quality of a cluster hierarchy. In “AbstractBehaviour of Distributed Executions with Applications to Visualization,”Ph.D. thesis, Technische Hochschule Darmstadt, Darmstadt, Germany, May1994, by T. Kunz, the author evaluates the quality of his tool bymeasuring the cohesion, which though expressed quantitatively isactually a qualitative measurement (the higher the better) within acluster and the coupling, a qualitative measure of the informationclusters must know about each other (the higher the worse), betweenclusters. For a cluster P of m processes, cohesion is quantified by:${{Cohesion}(P)} = \frac{\sum\limits_{i < j}{{Sim}_{f}\left( {p_{i},p_{j}} \right)}}{{m\left( {m - 1} \right)}/2}$

[0427] where Sim_(f) (P₁, P₂) is a similarity metric that equals:${Sim}_{f} = \frac{{{{A{\langle{\hat{C}}_{P_{1}}}}}{\hat{C}}_{P_{2}}}\rangle}{{{\hat{C}}_{P_{1}}} \cdot {{\hat{C}}_{P_{2}}}}$

[0428] Here, <â|{circumflex over (b)}> denotes the scaler product ofvectors â and {circumflex over (b)}, and ∥â∥ denotes the magnitude ofvector â. C_(P1) and C_(P2) are process characteristic vectors—in them,each element contains a value between 0 and 1 that indicates howstrongly a particular characteristic manifests itself in each process.Characteristics can include keywords, type names, function references,etc. A is a value that equals 1 if any of the following apply:

[0429] P₁ and P₁ are instantiations of the same source.

[0430] P₁ and P₂ are unique instantiations of their own source.

[0431] P₁ and P₂ communicate with each other.

[0432] A equals 0 if none of these is true (e.g., P₁ and P₂ arenonunique instantiations of separate source that do not communicate witheach other). Coupling is quantified by:${C\quad o\quad u\quad p\quad l\quad i\quad n\quad {g(P)}} = \frac{\underset{ij}{\sum\quad}{S\quad i\quad {m_{f}\left( {p_{i},q_{j}} \right)}}}{m\quad n}$

[0433] where q_(j)εQ, Q is the complement of P, and n=|Q|. The qualityof a cluster is quantified as its Coupling minus its Cohesion. In manycases, these metrics match many of the characteristics that intuitivelydifferentiate good and poor clusters, as shown in FIGS. 38A, B, and C.With reference to FIGS. 38A and C, Cohesion is high where clusterscorrespond to heavy communication and where clusters correspond toprocesses instantiated from the same source code. Coupling is shown tobe low in each of the above cases. With reference to FIG. 38B, Couplingis high when clusters do not correspond to heavily communicatingprocesses or to instances of the same source code. It is not clear,however, that the cluster in FIG. 38C should be assigned the samequality value as the cluster in FIG. 38A. Using these metrics, Kunzachieved qualities of between :15 and :31 for his clustering techniques.However, it is hard to tell what this means in terms of clusterusefulness.

[0434] 3. State-Based Debugging

[0435] State-based debugging techniques focus on the state of the systemand the state changes caused by events, rather than on eventsthemselves. The familiar source-level debugger for sequential programdebugging is state-based. This source-level debugger lets designers setbreakpoints in the execution of a program, enabling them to investigatethe state left by the execution to that point. This source-leveldebugger also lets programmers step through a program's execution andview changes in state caused by each step.

[0436] Concurrent systems have no unique meaning for an instant inexecution time. Stopping or single-stepping the whole system canunintentionally, but substantially, change the nature of interactionsbetween processes.

[0437] A. Consistent Cuts and Global State

[0438] In distributed event-based debugging, the concept of causality istypically of such importance that little of value can be discussedwithout a firm understanding of causality and its implications. Indistributed state-based debugging, the concept of a global instant intime is equally important.

[0439] Here again, it may seem intuitive to consider real-time instantsas the global instants of interest. However, just as determining thereal-time order of events is not practical or even particularly useful,finding accurate real-time instants makes little sense. Instead, aglobal instant is represented by a consistent cut. A consistent cut is acut of an event dependency graph representing an execution that (a)intersects each process exactly once and (b) points all dependenciescrossing the cut in the same direction. Like real-time instants,consistent cuts have both a past and a future. These are the subgraphson each side of the cut.

[0440]FIG. 39 shows that consistent cuts can be represented as a jaggedline across the space/time diagram that meets the above requirements.With reference to FIG. 39, a space/time graph 3900 is shown having afirst cut 3902 and a second cut 3904. All events to the left of eitherfirst cut 3902 or second cut 3904 are in the past of each cut, and allevents to the right are in the future of each cut, respectively. Firstcut 3902 is a consistent cut because no message travels from the futureto the past. Second cut 3904, however, is not consistent because amessage 3906 travels from the future to the past.

[0441]FIGS. 40A, B, and C show that a distributed execution shown in aspace/time diagram 4000 can be represented by a lattice of consistentcuts 4002, in which

is the start of the execution and ⊥ is system termination. Withreference to FIGS. 40A, B, and C, lattice of consistent cuts 4002represents the global statespace traversed by a single execution. Sincelattice of consistent cuts 4002's size is on the order of |E|^(|P|), it,unlike space/time diagrams, is never actually constructed. In theremainder of this chapter, to describe properties of consistent cutlattices, the symbol

relates cuts such that one immediately precedes the other and

relates cuts between which there is a path.

[0442] B. Single Stepping in a Distributed Environment

[0443] Controlled stepping, or single-stepping, through regions of anexecution can help with an analysis of system behavior. The programmercan examine changes in state at the completion of each step to get abetter understanding of system control flow. Coherent single-steppingfor a distributed system requires steps to align with a path through anormal execution's consistent cut lattice.

[0444] DPD works with standard single-process debuggers (called clientdebuggers), such as DBX, GDB, etc. Programmers can use these tools toset source-level break-points and single-step through individual processexecutions. However, doing so leaves the other processes executingduring each step, which can yield unrealistic executions.

[0445] Zernic gives a simple procedure for single-stepping using apost-mortem traversal of a consistent cut lattice. At each point in thestep process, there are two disjoint sets of events: the past set, orevents that have already been encountered by the stepping tool, and thefuture set, or those that have yet to be encountered. To perform a step,the debugger chooses an event, e,, from the future such that any eventsit depends on are already in the past, i.e., there are no future events,e_(f), such that e_(f)

e_(i). This ensures that the step proceeds between two consistent cutsrelated by

. The debugger moves this single event to the past, performing anynecessary actions.

[0446] To allow more types of steps, POET's support for single-steppinguses three disjoint sets: executed, ready, and nonready. The executedset is identical to the past set in “Using Visualization Tools toUnderstand Concurrency,” by D. Zernik, M. Snir, and D. Malki, IEEESoftware 9, 3 (1992), pp. 87-92. The ready set contains all events thatare fully enabled by events in the future, and the contents of thenonready set have some enabling events in either the ready or nonreadysets. Using these sets, it is possible to perform three different typesof steps: global-step, step-over, and step-in. Global-step and step-overmay progress between two consistent cuts not related

(i.e., there may be several intermediate cuts between the step cuts).

[0447] A global-step is performed by moving all events from the readyset into the past. Afterwards, the debugger must move to the ready setall events in the nonready set whose dependencies are in the executedset. A global-step is useful when the programmer wants information abouta system execution without having to look at any process in detail.

[0448] The step-over procedure considers a local, or single-process,projection of the ready and nonready sets. To perform a step, it movesthe earliest event from the local projections into the executed set andexecutes through events on the other processes until the next event inthe projection is ready. This ensures that the process in focus willalways have an event ready to execute in the step that follows.

[0449] Step-in is another type of local step. Unlike step-over, step-indoes not advance the system at the completion of the step; instead, thesystem advance is considered to be a second step. FIGS. 41A, B, C, and Dshow a space/time diagram before a step 4100 and a resulting space/timediagram after performing a global-step 4102, a step-over 4104, and astep-in 4106.

[0450] C. Runtime Consistent Cut Algorithms

[0451] It is occasionally necessary to capture consistent cuts atruntime. To do so, each process performs some type of cut action (e.g.,state saving). This can be done with barrier synchronization, whicherects a temporal barrier that no process can pass until all processesarrive. Any cut taken immediately before, or immediately after, thebarrier is consistent. However, with barrier synchronization, someprocesses may have a long wait before the final process arrives.

[0452] A more proactive technique is to use a process called the cutinitiator to send perform-cut messages to all other system processes.Upon receiving a perform-cut message, a process performs its cut action,sends a cut-finished message to the initiator, and then suspends itself.After the cut initiator receives cut-finished messages from allprocesses, it sends each of them a message to resume computation.

[0453] The cut obtained by this algorithm is consistent: no process isallowed to send any messages from the time it performs its own cutaction until all processes have completed the cut. This means that nopost-cut messages can be received by processes that have yet to performtheir own cut action. This algorithm has the undesirable characteristicof stopping the system for the duration of the cut. The followingalgorithms differ in that they allow some processing to continue.

[0454] 1. Chandy-Lamport Algorithm

[0455] The Chandy-Lamport algorithm does not require the system to bestopped. Once again, the cut starts when a cut initiator sendsperform-cut messages to all of the processes. When a process receives aperform-cut message, it stops all work, performs its cut action, andthen sends a mark on each of its outgoing channels; a mark is a specialmessage that tells its recipient to perform a cut action before readingthe next message from the channel. When all marks have been sent, theprocess is free to continue computation. If the recipient has alreadyperformed the cut action when it receives a mark, it can continueworking as normal.

[0456] Each cut request and each mark associated with a particular cutare labeled with a cut identifier, such as the process ID of the cutinitiator and an integer. This lets a process distinguish between marksfor cuts it has already performed and marks for cuts it has yet toperform.

[0457] 2. Color-Based Algorithms

[0458] The Chandy-Lamport algorithm works only for FIFO (First In FirstOut) channels. If a channel is non-FIFO, a post-cut message may outrunthe mark and be inconsistently received before the recipient is evenaware of the cut, i.e., it is received in the cut's past. The remedy tothis situation is a color-based algorithm. Two such algorithms arediscussed below.

[0459] The first is called the two-color, or red-white, algorithm. Withthis algorithm, information about the cut state is transferred with eachmessage. Each process in the system has a color. Processes not currentlyinvolved in a consistent cut are white, and all messages transmitted aregiven a white tag. Again, there is a cut initiator that sendsperform-cut messages to all system processes. When a process receivesthis request, it halts, performs the cut action, and changes its colorto red. From this point on, all messages transmitted are tagged with redto inform the recipients that a cut has occurred.

[0460] Any process can accept a white message without consequence, butwhen a white process receives a red message, it must perform its cutaction before accepting the message. Essentially, white processes treatred messages as cut requests. Red processes can accept red messages atany time, without consequence.

[0461] A disadvantage of the two-color algorithm is that the system mustreset all of the processes back to white after they have completed theircut action. After switching back, each process must treat red messagesas if they were white until they are all flushed from the previous cut.After this, each process knows that the next red message it receivessignals the next consistent cut.

[0462] This problem is addressed by the three-color algorithm, whichresembles the two-color algorithm in that every process changes colorafter performing a cut; it differs in that every change in colorrepresents a cut. For colors zero through two, if a process with thecolor c receives a message with the color (c−1) mod 3, it registers thisas a message-in-flight (see below). On the other hand, if it receives amessage with the color (c+1) mod 3, it must perform its cut action andswitch color to (c+1) mod 3 before receiving the message. Of course,this can now be generalized to n-color algorithms, but three colors areusually sufficient.

[0463] Programmers may need to know about messages transmitted acrossthe cut, or messages-in-flight. In the two-color algorithm,messages-in-flight are simply white messages received by red processes.These can all be recorded locally, or the recipient can report them tothe cut initiator. In the latter case, each red process simply sends theinitiator a record of any white messages received.

[0464] It is not safe to switch from red to white in the two-coloralgorithm until the last message-in-flight has been received. This canbe detected by associating a counter with each process. A processincrements its counter for each message sent and decrements it for eachmessage received. When the value of this counter is sent to theinitiator at the start of each process's cut action, the initiator canuse the total value to determine the total number of messages-in-flight.The initiator simply decrements this count for each message-in-flightnotification it receives.

[0465] D. State Recovery—Rollback and Replay

[0466] Since distributed executions tend to be nondeterministic, it isoften difficult to reproduce bugs that occur during individualexecutions. To do so, most distributed debuggers contain a rollbackfacility that returns the system to a previous state. For this to befeasible, all processes in the system must occasionally save theirstate. This is called checkpointing the system. Checkpoints do not haveto save the entire state of the system. It is sufficient to save onlythe changes since the last checkpoint. However, such incrementalcheckpointing can prolong recovery.

[0467] DPD makes use of the UNIX fork system call to performcheckpointing for later rollback. When fork is called, it makes an exactcopy of the calling process, including all current states. In the DPDcheckpoint facility, the newly forked process is suspended and indexed.Rollback suspends the active process and resumes an indexed process. Theproblem with this approach is that it can quickly consume all systemmemory, especially if checkpointing occurs too frequently. DPD'ssolution is to let the programmer choose the checkpoint frequencythrough use of a slider in its GUI.

[0468] Processes must sometimes be returned to states that were notspecifically saved. In this case, the debugger must do additional workto advance the system to the desired point. This is called replay and isperformed using event trace information to guide an execution of thesystem. In replay, the debugger chooses an enabled process (i.e., onewhose next event has no pending causal requirements) and executes it,using the event trace to determine where the process needs to block fora message that may have arrived asynchronously in the originalexecution. When the process blocks, the debugger chooses the nextenabled process and continues from there. In this way, a replay iscausally identical to the original execution.

[0469] Checkpoints must be used in a way that prevents domino effects.The domino effect occurs when rollbacks force processes to restore morethan one state. Domino effects can roll the system back to the startingpoint. FIG. 42 shows a space time diagram 4200 for a system that issubject to the domino effect during rollback. With reference to FIG. 42,if the system requests a rollback to checkpoint c₃ 4202 of process P₃4204, all processes in the system must roll back to c₁ (i.e., roll backto P₃. c₂ 4206 requires a roll back to P₂. c₂ 4208, which requires aroll back to P₁. c₂ 4210, which requires a roll back to P₃. c₁ 4212,which requires a roll back to P₂. c₁ 4214, which requires a final rollback to P₁. c₁ 4216). The problem is caused by causal overlaps betweenmessage transfers and checkpoints. Performing checkpoints only atconsistent cuts avoids a domino effect.

[0470] E. Global State Predicates

[0471] The ability to detect the truth value of predicates on globalstate yields much leverage when debugging distributed systems. Thistechnique lets programmers raise flags when global assertions fail, setglobal breakpoints, and monitor interesting aspects of an execution.Global predicates are those whose truth value depends on the statemaintained by several processes. They are typically denoted with thesymbol φ. Some examples include (Σ₁c₁>20) and (c₁<20^ c₂ 5), where c_(i)is some variable in process P₁ that stores positive integers. In theworst case (such as when (Σ₁c₁>20) is false for an entire execution), itmay be necessary to get the value of all such variables in allconsistent cuts. In the following discussion, we use the notationC_(a)|=Φ to indicate that Φ is true in consistent cut C_(a).

[0472] At this point, it is useful to introduce branching time temporallogic. Branching time temporal logic is predicate logic with temporalquantifiers, P, F, G, H, A, and E. PΦ is true in the present if Φ wastrue at some point in the past; FΦ is true in the present if Φ will betrue at some point in the future; GΦ is true in the present if Φ will betrue at every moment in the future; and HΦ is true in the present if Φwas true at every moment of the past. Notice that GΦ is the same as

F

Φ, and HΦ is the same as

P

Φ.

[0473] Since global time passage in distributed systems is marked by apartially ordered consistent cut lattice rather than by a totallyordered stream, we need the quantifiers A, which precedes a predicatethat is true on all paths, and E, which precedes a predicate that istrue on at least one path. So, AFΦ is true in the consistent cutrepresenting the present if Φ is true at least once on all paths in thelattice leaving this cut. EPΦ is true in the consistent cut representingthe present if Φ is true on at least one path leading to this cut.

[0474] A monotonic global predicate is a predicate Φ such that C_(a)|=Φ

C_(a)|=AGΦ. A monotonic global predicate is one that remains true afterbecoming true. An unstable global predicate, on the other hand, is apredicate Φ such that C_(a)|=Φ

C_(a)|=EG

Φ. An unstable global predicate is one that may become false afterbecoming true.

[0475] 1. Detecting Monotonic Global Predicates

[0476] Monotonic predicates can be detected any time after becomingtrue. One algorithm is to occasionally take consistent cuts and evaluatethe predicate at each. In fact, it is not necessary to use consistentcuts, since any transverse cut whose future is a subset of the future ofthe consistent cut in which the predicate first became true will alsoshow the predicate true.

[0477] 2. Detecting Unstable Global Predicates

[0478] Detecting arbitrary unstable global predicates can take at worst|E|^(|P|) time, where |E|^(|P|) is the size of an execution's consistentcut lattice, [E] is the number of events in the execution, and [P] isthe number of processes. This is so, because it may be necessary to testfor the predicate in every possible consistent cut. However, there are afew special circumstances that allow |E| time algorithms.

[0479] Some unstable global predicates are true on only a few pathsthrough the consistent cut lattice, while others are true on all paths.Cooper and Marzullo describe predicate qualifiers definitely Φ forpredicates that are true on all paths (i.e.,

|=A FΦ) and possibly Φ for those that are true on at least one path(i.e.,

|=>E FΦ.

[0480] The detection of possibly Φ for weak conjunctive predicates, orglobal predicates that can be expressed as conjunctions of localpredicates, is Φ (|E|) The algorithm for this is to walk a path throughthe consistent cut lattice that aligns with a single process, P_(t),until either (1) the process's component of Φ is true or (2) there is noway to proceed without diverging from P_(t). In either case, the targetprocess is switched and the walk continued. This algorithm continuesuntil it reaches a state in which all components of the predicate aretrue or until it reaches ⊥. In this way, if there are any consistentcuts where all parts of the predicate simultaneously hold, the algorithmwill encounter at least one.

[0481] Detection of possibly Φ for weak disjunctive predicates, orglobal predicates that can be expressed as disjunctions of localpredicates, is also φ (|E|); it is the same algorithm as above, exceptit halts at the first node where any component is true. However, weakconjunctive and disjunctive predicates constitute only a small portionof the types of predicates that could be useful in debugging distributedsystems.

[0482] 4. Conclusions

[0483] Complicating the debugging of heterogenous embedded systems aredesigns composed of concurrent and distributed processes. Most of thedifficulty in debugging distributed systems results from concurrentprocesses with globally unscheduled and frequently asynchronousinteractions. Multiple executions of a system can produce wildly varyingresults—even if they are based on identical inputs. The two maindebugging approaches for these systems are event based and state based.

[0484] Event-based approaches are monitoring approaches. Events arepresented to a designer in partially ordered event displays, calledspace/time displays. These are particularly good at showinginter-process communication over time. They can provide a designer withlarge amounts of information in a relatively small amount of space.

[0485] State-based approaches focus locally on the state of individualprocesses or globally on the state of the system. Designers can observeindividual system states, set watches for specific global predicates,step through executions, and set breakpoints based on global statepredicates. These approaches deal largely with snapshots, consideringtemporal aspects only as differences between snapshots.

[0486] As distributed systems increase in size and complexity, the sheervolume of events generated during an execution grows to a point where itis exceedingly difficult for designers to correctly identify aspects ofthe execution that may be relevant in locating a bug. For distributedsystem debugging techniques to scale to larger and faster systems,behavioral abstraction will typically become a necessity to helpdesigners identify and interpret complicated behavioral sequences in asystem execution. Finally, embedded systems must execute in a separateenvironment from the one in which they were designed and embeddedsystems may also run for long periods of time without clear stoppingpoints. Debugging them requires probes to report debugging informationto a designer during the execution. These probes inevitably alter systembehavior, which can mask existing bugs or create new bugs that are notpresent in the uninstrumented system. While it is not possible tocompletely avoid these probe effects, they can be minimized throughcareful placement, or masked through permanent placement.

[0487] Evolution Diagrams

[0488] Evolution diagrams are visual representations of a system'sbehavior over time. They resemble the space/time diagrams discussed inChapter 2, but they explicitly include information related to componentbehavior and changes in behavior caused by events. Evolution diagramstake advantage of the exposure provided by coordination interfaces topresent more complete views of system executions than can be obtainedfrom space/time diagrams.

[0489] Evolution diagrams explicitly show events, message trafficbetween components, changes in component behavior, and correlationsbetween local behavior changes. Through them, designers can easily spottransaction failures and components operating outside of their expectedmodel. Essentially, evolution diagrams are event graphs interpreted inthe context of the system being debugged. While evolution diagrams donot aid the debugging of individual lines of action code, they can helpdesigners pinpoint the specific action to debug. In Section 4.5.2, wediscuss how source-level debuggers can be integrated coherently withevolution diagrams.

[0490]FIG. 43 portrays the evolution of a dedicated RPC transaction. Ithas traces for both components 4310 (bars enclosed by dashed lines),interface states 4312 (shown by solid horizontal bars), and events 4314(shown by vertical ovals spanning the space affected by the event). Thefigure displays all essential aspects of an RPC transaction.

[0491] The remainder of this section describes event and staterepresentations, event dependencies, the use of evolution diagrams withhigh-level simulation to detect transaction failures and inappropriatecomponent behavior, and debugging issues that become evident atsynthesis (“synthesis effects”).

[0492] Event Representations

[0493] Event representations display all event types described earlier(e.g., transmission and reception of data, changes in control state, andchanges in more general shared state). Event representations have a nameand can also have a visual cue, or icon, such as those shown in FIG. 44.

[0494] The design methodology described above clearly identifies thetypes of events that can be generated by each component. Although thereare many more specific types than shown in FIG. 44, each specific typemust be derived from one of those shown. For example, bothRPC.Client.argument.send and RPC.server.return.send are derived from theprimitive send event.

[0495] State Representations

[0496] Modes and other types of state are displayed as horizontal barsannotated with the name of the state and, where appropriate, the value.These bars extend across the duration of the mode or the value. See FIG.43. Through state views, designers can monitor component behaviorchanges over time.

[0497] The types of state that can be displayed are the values ofexported variables and control state. FIG. 45 shows illustrativeprimitive state types.

[0498] Event Dependencies

[0499] Events on different components may be connected explicitly bymessages traveling between them or implicitly by coordinator constraintsand actions, as described earlier. Explicit connections are displayed asarrows between transmit and receive events. Implicit connections aredisplayed as diagonal lines without arrows, where the event on the leftside is the immediate predecessor of the event on the right side. Theseconnections indicate dependencies in the underlying event graph. See thediscussion above regarding FIG. 25, et seq.

[0500] Debugging with Evolution Diagrams

[0501] Evolution diagrams can be integrated with high-level simulation,letting designers fix many bugs before synthesis and mapping to ahardware architecture. FIG. 46 shows the evolution diagram of a correctlookup and an initiate call transaction for the mobile telephoneexample. This diagram shows the top level of the design hierarchy forthe cell phone itself. The transaction begins when a user looks up anumber in the address book. When the number is selected, the GUI sendsit to the connection subsystem with a send action, which generates asend event 4604. It then waits for the connection to go through. Thisdemonstrates the use of design hierarchy to hide information; manyaspects of the GUI and connection subsystem's interaction are hiddenfrom view, but the information presented gives a general idea about whatis going on inside the system. The connection subsystem then contactsthe call recipient and starts the ring signal in the voice subsystem4608. When the recipient answers the phone, the GUI, the connectionsubsystem, and the voice subsystem all switch to the call-in-progressmode 4610 and begin the corresponding behavior.

[0502] If part of the transaction fails (for example, if the phone neverplays a ring-back tone), the designer can often find the source of theproblem in an evolution diagram. If the problem is caused by a controlfailure, the voice subsystem does not enter ringing mode, and designerswould see something like the evolution diagrams shown in FIG. 47. FIG.47A shows the results of a local control failure, where the voicesubsystem receives the appropriate trigger 4710 but does not respondappropriately by moving into ring mode. FIG. 47B shows the sametransaction, except that the problem is in the distributed controlstructure.

[0503] Selective Focus in Debugging

[0504] Selective focus describes techniques by which designers can limitthe data presented based on its relevance at any point in time.Selective focus plays an important role in debugging with evolutiondiagrams. For example, designers begin debugging the 47 problem needingonly high-level information about the system's behavior. Once thehigh-level source of the problem is found, designers can descend thedesign hierarchy to pinpoint the cause. FIG. 48 shows an expanded viewof the voice subsystem, where, at the point of failure, the soundcoordinator fails to switch on the “ringing” mode. A designer can nowinvestigate the specific action responsible for this. With selectivefocus, the designer can quickly bore into the affected region to helpidentify the bug's cause.

[0505] Selective focus is also useful in debugging problems with layeredcoordination. Recall from FIG. 19 that there are several coordinationlayers between the call managers on the cell phone and the switchingcenter, but that there is conceptual coordination between peer layers.This conceptual coordination enables a form of selective focus.

[0506] Consider an example where designers discover that a cell phonedrops calls at random moments. Using standard good troubleshootingprocedure, they begin debugging at the layer nearest the detectedproblem: call management. Using evolution diagrams and selective focus,the designers can investigate the bug on the call management layerwithout requiring details from lower layers. They can review theprogress of the phone call up until the moment of the drop.

[0507] Assume the cause of the bug is elsewhere (for example, the radioresource layer sometimes fails when performing handoffs between cells),finding no specific problem in the call management layer, designers canproceed down the protocol stack to the mobility management layer and,finding no problem there, move on to the radio resource layer. At theradio resource layer, designers will find that, at the time of the dropoff, the radio resource component was in the midst of executing ahandoff, wherein the problem lies. Thus, they may immediately suspectthat the cause of the problem was related to the handoff.

[0508] Correlating Disparate Behaviors

[0509] Consider a bug that manifests itself in interactions amongseveral components. FIG. 49 shows the source code for components X, Y,and Network Interface, shading the aspects that participate in the bug.As shown, these participating sections of code are scattered acrossthree files. Because of this scattering, and because the relevantsections of code do not execute at the same time, a designer is unlikelyto spot the bug easily through source-level debugging techniques.

[0510]FIG. 50 shows an evolution diagram of a bug from a system builtwith the present methodology. At the point of failure, the evolutiondiagram shows that the network interface's resource is taken, and that Xis clearly holding the resource. Scrolling back, we find that when Ytried to obtain the resource, it could not because X was still holdingit. Scrolling back further, we may find a number of such repeatedattempts, each with similar failures. And finally, scrolling back to thebeginning, we find that grabbed the resource before Y's first attempt,and that never released it. We now know that X was the primary cause ofthe problem, and the problem is now reduced to debugging a singlecomponent.

[0511] Event Persistence

[0512] In each of the examples described above, it was necessary toreview portions of the system execution several times to track down abug. For the ring failure bug, we needed to review the failure to obtaindetailed information about the voice component's behavior. For the calldropping bug, we needed to review the execution at least three times totrace the bug through the call management, mobility management, andradio resource layers. For the resource allocation bug, we needed toexamine the behavior of component X in the vicinity of the bug todetermine why it never released the resource. Repeated executions of aconcurrent system with the same inputs can produce greatly varyingresults. Specific interactions may differ on each new execution,preventing designers from making progress in debugging.

[0513] To avoid this, and ensure that each execution is identical to thelast, it is necessary to: (1) maintain a store of an execution's eventsand the relationships among them, and (2) provide our debugging toolswith the means to traverse this store many times with differingperspectives. We can operate directly on this store, as described later.

[0514] Synthesis Effects

[0515] Designers inevitably make assumptions about relative timing thatare not necessarily borne out by the implementation. Unfortunately,idealized simulation without regard for architectural issues can givedesigners a skewed perspective. It is only at synthesis that the actualtiming between events and the relative timing between actions congeals.Solidifying timing concerns affects event orders and timing constraints.The synthesized system must be tested and validated by a designer.

[0516] An example of synthesis effects can be seen in the use of theround-robin/preempt coordinator described earlier. In this protocol,components usually take turns with the resource, but one of thecomponents is a preemptor that can take over without waiting for itsturn. A potential source of problems is that after preemption, controlalways returns to the component that follows the preemptor in the ring,not to the component that was preempted. This may still be a reasonabledesign decision, since distributed tracking of the preempted componentcan be expensive. However, in the mapping shown in FIG. 51, thiscombined protocol performs poorly.

[0517] As shown in FIG. 52A, before synthesis, this system approximatesfair access to the resource. After synthesis, as shown in FIG. 52B,preemption seems to occur more frequently than expected. This results incomponents C, D, and E getting few or no chances to access the resource.In this case, a bug fix may be to alter the protocol so that it tracksthe component that held the resource before preemption, to try differentmappings to alter the situation so that preemption occurs lessfrequently, or to try different protocols altogether.

[0518] Behavioral Perspectives

[0519] Design hierarchy is a very important part of managing debuggingcomplexity: it allows designers to observe what is happening in a systemor component at a general level, and then further refine the view.Unfortunately, clusters that make sense for design purposes are notalways the ones needed for debugging. FIG. 53A shows part of the designdecomposition of the GUI module. To compare numbers generated by thekeypad with packets sent out through the transport, designers need onlybe able to put the relevant parts into focus and represent the rest ofthe system as a cluster, as shown in FIG. 53B.

[0520] Behavioral perspectives allow designers to tailor selective focusfor their convenience. A behavioral perspective is a set of clusters andfilters, some of which may be derived from the design hierarchy, whileothers may be specified by a designer when the design hierarchy is notsufficient. Special-purpose clusters and filters are described below.

[0521] Special-purpose Clusters

[0522] Designers use special-purpose clusters to help reduce the amountof clutter presented in a display without eliminating sources ofinformation. There are three types of special-purpose clusters:component, event, and state. Component clusters combine severalcomponent traces into one; event clusters combine sequences of events ona single component into one; and state clusters combine several statetraces into one. Designers can form clusters that are separate from thedesign hierarchy, as shown in FIG. 54.

[0523] Clusters can be described in two ways: visually, throughselection on an evolution diagram, or textually, through cluster lists(see Listing number 1, as follows): Listing 1: ClusterComponent “C1, C2”{ C1, C2 } ClusterStates C1. “Z” { C1. Q, C1. P }

[0524] Special-purpose Filters

[0525] Filters remove specific events, states, and even components froma designer's view. Using filters, designers can observe only the partsof an execution that pertain to a specific debugging objective. Filterswork well with clusters to help a designer reduce the total noise in anevolution diagram.

[0526] Like clusters, filters can be described both visually andtextually. Filter lists can have the form ALL except <event list>. Thus,in cases where there are more event types to be filtered than passed, adesigner can use the filter lists to specify only those events thatshould be shown.

[0527]FIGS. 55A and 55B show before and after snapshots, respectively,of an evolution diagram using the filter description shown in Listingnumber 2, as follows: Listing 2: Filter { Components { C3 } States { ALLexcept C1.Q, C2.S } Events { ALL except C1.a.send, C1.r.rec, C2.a.rec,C2.r. send } }

[0528] Event type names in this listing have the form:

[0529] component_name.interface.specific_type.

[0530] The result of applying this filter clarifies an RPC-like aspectof this coordination. Designers can also use filters to expose eventsand states that are not normally visible at a particular level of focus.

[0531] It will be apparent to those having skill in the art that manychanges may be made to the details of the above-described embodiment ofthis invention without departing from the underlying principles thereof.The scope of the present invention should, therefore, be determined onlyby the following claims.

1. A method for debugging a distributed software system that includesfirst and second components and a coordinator that implements a desiredinteraction between the first and second components; each component andthe said coordinator including coordination interfaces that exposeevents, and the debugging method comprising the steps of: creating anevent record in response to each exposed event that occurs during anexecution of the distributed software system, each event recordincluding identification of a component that generated the event and alocal time stamp; accumulating the event records into an event database;and then displaying an evolution diagram for use by a developer indebugging the distributed software system, the evolution diagramincluding a graphical representation of at least a selected portion ofthe event database.
 2. A method according to claim 1 wherein saiddisplaying step includes displaying at least a first selected eventtogether with the identification of the component that generated theselected event.
 3. A method according to claim 1 wherein the displayingstep includes presenting in the evolution diagram: a first graphicaldisplay element representing a first component of the software system;and a second graphical display element representing a first selectedevent generated by the first component; the first and second graphicaldisplay elements being juxtaposed so as to visually indicate that thefirst selected event was generated by the first component.
 4. A methodaccording to claim 1 wherein the displaying step includes presenting inthe evolution diagram: a first graphical display element representing afirst event; a second graphical display element representing a secondevent; and a third graphical display element indicating a causalrelationship between the first and second events.
 5. A method accordingto claim 4 wherein each of the first and second graphical displayelements comprises an identifier of a component that generated thecorresponding event.
 6. A method according to claim 4 wherein the causalrelationship comprises sending a message as the first event andreceiving the said message as the second event.
 7. A method according toclaim 6 wherein the third graphical display element comprises an arrowextending from the first graphical display element to the secondgraphical display element.
 8. A method according to claim 4 wherein atleast one of the first and second events consists of a control statechange exposed at the coordination interface of one of the components.9. A method according to claim 4 wherein at least one of the first andsecond events consists of a message sent from a port at the coordinationinterface one of the components.
 10. A visual display for use by adeveloper in debugging a distributed or concurrent software system, thevisual display comprising an evolution diagram responsive to apredetermined set of event records generated during an execution of thesubject software system; each event record reflecting a correspondingsoftware system event.
 11. A visual display according to claim 10wherein each of the event records is responsive to one of a timer tick,a data departure, a data arrival or a mode change.
 12. A visual displayaccording to claim 10 wherein the evolution diagram includes: a firstgraphical display element representing a first component of the softwaresystem; and a second graphical display element representing a firstevent generated by the first component; the first and second graphicaldisplay elements being juxtaposed in the visual display so as to cue thedeveloper that the first event was generated by the first component. 13.A visual display according to claim 12 wherein the first graphicalelement representing a first component of the software system comprisesa generally horizontal bar and the second graphical display elementcomprises a generally vertical icon overlapping the horizontal bar. 14.A visual display according to claim 10 wherein the evolution diagramincludes: a first graphical display element representing a firstcomponent of the software system; a second graphical display elementrepresenting a second component of the software system; and a thirdgraphical display element representing an implicit message from thefirst component to the second component.
 15. A visual display accordingto claim 10 wherein the evolution diagram includes: a first graphicaldisplay element representing a first component of the software system; asecond graphical display element representing a second component of thesoftware system; and a third graphical display element representing anexplicit message from the first component to the second component.
 16. Avisual display according to claim 10 wherein the evolution diagramincludes: a first graphical display element representing a first event;a second graphical display element representing a second event; and athird graphical display element representing a causal relationshipbetween the first and second events.
 17. A visual display according toclaim 16 wherein the first graphical display element includesidentification of a component that generated the said first event; thefirst event is sending a message; the second event is receipt of amessage caused by the first event; and the third graphical displayelement comprises an arrow having a tail positioned adjacent the firstgraphical display element and a head positioned adjacent the secondgraphical display element.
 18. A visual display according to claim 12wherein the first graphical display element includes indicia identifyinga control state of the first component and indicating its current value.19. A visual display according to claim 12 wherein the first graphicaldisplay element includes indicia identifying an exported variable of thefirst component and indicating its current value.
 20. A method fordebugging a distributed, hierarchical software system that includes aplurality of design levels, each design level comprising two or morecomponents and a coordinator that implements desired interaction betweenthe said components; each component and the coordinators with which itinteracts including respective complementary coordination interfacesthat expose events, and the debugging method comprising the steps of:creating an event record in response to each exposed event that occursduring an execution of the distributed software system, each eventrecord including identification of a component that generated the eventand a local time stamp; accumulating the event records into an eventdatabase; partially ordering the event database based on the timestamps; and then displaying an evolution diagram for use by a developerin debugging the distributed software system, the evolution diagramincluding a graphical representation of at least a selected portion ofthe event database.
 21. A method according to claim 20 wherein saiddisplaying step includes selecting one of the design levels anddisplaying an evolution diagram corresponding to the selected designlevel including graphical indicia including events exposed at thecoordination interfaces of components defined at the selected designlevel, thereby hiding subsystem interactions from view.
 22. A methodaccording to claim 21 and further comprising selecting a temporal subsetof the event database for inclusion in the displayed evolution diagramthereby selectively focusing on a region of interest to the developer.23. A method according to claim 20 wherein the execution is simulated.24. A method according to claim 20 wherein the execution is carried outon a target hardware platform.
 25. A method according to claim 20wherein the software system is instrumented so as to generate eventrecords at selected points in execution that are not events exposed atthe coordination interface of a component.
 26. A method according toclaim 20 wherein the execution is carried out on a distributed, embeddedtarget hardware platform comprising a plurality of hardware subsystems,and the event records are collected from each hardware subsystem.
 27. Amethod according to claim 20 wherein said displaying the evolutiondiagram includes combining a selected sequence of events on a singlecomponent so as to form an event cluster, and then displaying the eventcluster.
 28. A method according to claim 20 wherein said displaying theevolution diagram includes combining a selected group of components soas to form a component cluster, and then displaying a single tracerepresenting the component cluster.
 29. A method according to claim 20wherein said displaying the evolution diagram includes combining aselected group of state traces so as to form a state cluster, and thendisplaying a single trace representing the state cluster.
 30. A methodaccording to claim 20 and further comprising filtering selected events,states or components from the evolution diagram.